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Preface 


Portable  maintenance  aids  (PMAs)  are  being  developed  to 
access  and  store  electronic  technical  information  on  the 
flightline.  The  current  prototype  PMA  has  redundant  access/ 
navigational  features.  The  purpose  of  this  research  was  to 
determine  the  best  access /navigational  features  installed  on 
the  PMA. 
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advisors.  Major  Michael  Morabito  and  Professor  Art  Munguia. 
Their  contributions  were  immeasurable.  We  were  lucky  to 
have  thesis  advisors  who  fit  our  temperament  and  style. 

They  were  always  available  for  suggestions  and  provided 
valuable  input  when  we  needed  it;  but  most  importantly,  they 
left  us  alone  when  we  needed  it  most. 

We  would  also  like  to  thank  the  folks  at  Armstrong 
Laboratory  (AL)  for  sponsoring  us  and  providing  insight  and 
guidance.  Many  thanks  to  two  individuals  at  AL,  Barbara 
Masquelier  and  Laurie  Quill.  Without  the  help  of  these  two 
professionals,  we  could  not  have  accomplished  this  thesis. 
Laurie's  expert  knowledge  on  graphical  user  interfaces  (GUI) 
was  instrumental  in  developing  our  experimental  design  and 
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ensuring  our  screens  conformed  to  GUI  standards.  A  special 
thanks  to  Barbara/  who  graciously  agreed  to  take  us  under 
her  wing,  so  soon  after  helping  two  92S  thesis  jgroups.  We 
thank  her  for  her  experience,  knowledge,  energy,  motivation, 
and  sense  of  humor. 

Of  course,  we  couldn't  have  completed  our  thesis 
without  the  help  of  the  people  who  graciously  consented  to 
participate  in  our  experiment.  Thanks  to  our  classmates, 
instructors  and  everyone  who  participated  in  our  pilot 
study.  Their  help  gave  us  valuable  insight  and  helped  us 
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participated  in  our  experiment.  Without  these  folks,  our 
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The  AFIT  experience  was  certainly  much  more  enjoyable 
because  of  the  people  we  met  while  we  were  here.  Thanks  to 
our  friends,  especially  the  bowling  and  Friday  Nite  dinner 
crowd.  Last,  but  certainly  not  least,  is  a  special  thanks 
to  Lisa's  husband,  Steve.  Steve  endured  endless  hours  of 
thesis — hogging  the  computer  (he  was  also  trying  to  complete 
a  thesis)  and  eating  all  the  food. 
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Abstract 

Portable  maintenance  aids  (PMAs)  are  being  developed  to 
access  and  store  electronic  technical  information  on  the 
flightline.  The  prototype  PMA  designed  by  Armstrong 
Laboratory  personnel  has  redundant  access/navigational 
features.  Redundancy  increases  software  memory  usage  and 
adds  unnecessary  weight  to  the  PMA.  The  purpose  of  this 
research  was  to  determine  the  best  access/navigational 
feature  installed  on  the  PMA.  The  best  feature  is  the 
feature  that  provides  the  highest  degree  of  user 
satisfaction.  An  experiment  was  conducted  to  evaluate  the 
following  features  for  screen  and  menu  access  and 
navigation,  rospectively :  dedicated/hardware  keys, 
programmable  soft  keys,  and  push  button  keys;  and  number 
keys,  cursor  control  keys,  and  programmable  soft  keys. 
Modified  computer  screens  from  the  prototype  PMA  were  used 
on  a  laptop  personal  computer,  which  simulated  the  PMA,  to 
evaluate  each  feature  one-at-a-time .  Twenty-eight 
maintenance  technicians  located  at  Wright-Patterson  AFB,  OH, 
rated  and  ranked  each  access/navigational  feature.  The 
results  indicated  that  the  best  feature  for  screen  access 
and  navigation  was  the  dedicated/hardware  keys  and  the  best 
feature  for  menu  access  and  navigation  was  the  niimber  keys. 
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EVALUATION  OF  ACCESS/NAVIGATIONAL 
FEATURES  OF  A  GRAPHICAL  USER  INTERFACE 
INSTALLED  ON  A  PORTABLE  MAINTENANCE  AID 

I.  Introduction 

This  chapter  introduces  the  general  problem  of  the 
usefulness  of  access/navigational  features  of  a  graphical 
user  interface  (GUI)  installed  on  a  portable  maintenance  aid 
(PMA) — a  lightweight,  hand-held  computer  used  on  the 
flightline  to  access  aircraft  technical  data  and  diagnostic 
aids.  Although  past  studies  have  shown  the  usefulness  of  a 
PMA,  very  little  has  been  done  to  evaluate  the  interface 
installed  on  the  computer  (Masguelier,  1991;  Friend  and 
Grinstead,  1992). 

In  this  chapter,  we  examine  the  need  for  research,  and 
the  general  issues  that  affect  the  problem.  We  also  examine 
the  Computer-aided  Acquisition  and  Logistic  Support  (CALS) 
initiative.  Additionally,  we  give  a  brief  overview  of  GUIs 
and  interactive  electronic  technical  manuals  (lETMs).  Then 
we  define  the  specific  problem,  the  research  objectives,  the 
investigative  questions,  and  the  scope  and  limitations  of 
the  research.  Lastly,  we  summarize  the  need  and  importance 
to  evaluate  access/navigational  features  of  a  GUI  installed 
on  a  PMA,  and  give  an  overview  of  the  remaining  chapters. 
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As  the  complexity  of  weapon  systems  increases,  so  does 
the  complexity  of  the  maintenance  support  equipment.  An 
example  of  such  support  equipment  is  the  PMA.  Past  studies 
have  demonstrated  the  usefulness  of  a  PMA.  Armstrong 
Laboratory  (AL)  Logistics  Research  Division,  formerly  known 
as  Air  Force  Human  Resources  Laboratory,  demonstrated  that  a 
PMA  would  be  useful  in  a  field  level  maintenance  shop 
(Thomas  and  Clay,  1988).  A  later  study  by  Jeffrey  Friend 
and  Randy  Grinstead  compared  two  different  PMAs — a  head 
mounted  display  device  and  a  flat  screen  display  device 
(Friend  and  Grinstead,  1992).  Although  various  studies  have 
evaluated  the  usefulness  of  PMAs,  they  have  not  evaluated 
the  usefulness  of  GUIs. 

Gerald  Streff  and  Robert  Gundel  conducted  a  study 
comparing  the  advantages  of  access  methods — joystick  versus 
the  four-cursor  keys — in  using  the  GUI  installed  on  the  PMA 
(Streff  and  Gundel,  1992).  They  statistically  concluded 
that  there  were  little  differences  between  the  effectiveness 
of  either  input  device.  Additionally,  Michael  Morris  in 
1990  conducted  a  broader  study  on  the  efficiency  and 
effectiveness  of  GUIs  comparing  them  against  text  and 
command  base  interfaces.  He  concluded  that  "graphical  user 
interfaces  offer  significant  advantages...  graphical 
interface  systems  required  significantly  less  time  to  learn” 
(Morris,  1990:93).  Armstrong  Laboratory  personnel  decided 
to  use  a  GUI  on  the  prototype  PMA. 
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According  to  Barbara  Masguelier,  Human  Factors  Engineer 


for  AL,  lab  personnel  performed  various  studies  with  the 
PMA.  From  these  studies,  lab  personnel  gained  knowledge  on 
the  user  and  environment  which  enabled  them  to  design  the 
PMA  graphical  user  interface.  The  GUI  design  criteria 
centered  around  the  following: 

1.  Limiting  the  amount  of  computer  training  needed. 

2.  Meeting  user  expectations. 

3.  Providing  a  consistent  GUI  among  hardware 
platforms . 

4.  Increasing  design  flexibility  and  creating  added 
capability.  (Masguelier,  1992) 

However,  there  still  exists  a  need  to  test  the  usefulness  of 
access/navigational  features  of  the  GUI  before  future  PMAs 
are  fielded.  This  has  major  implications  because  PMAs  are 
expected  to  be  implemented  in  future  weapon  systems  (e.g., 
F-22  and  B-2  programs). 

General  Issues 

The  Computer-aided  Acguisition  and  Logistics  Support 
(CALS)  initiative  led  to  the  development  of  specifications 
for  interactive  electronic  technical  manuals  (lETMs).  The 
PMA  software  is  governed  by  a  specification  on  lETMs  which 
will  be  discussed  later  in  this  section.  To  better 
understand  the  need  for  research,  a  brief  overview  is 
provided,  including  a  discussion  of  CALS,  GUIs,  and  lETMs. 

CALS.  The  Computer-aided  Acguisition  and  Logistic 
Support  initiative  began  in  1984  to  apply  existing  and 


emerging  computer  technologies  to  the  acquisition  and 

support  of  weapons  systems.  CALS  will  enable  OoD  to: 

Automate  the  storage,  capture,  exchange  and  retrieval 
of  contracting,  engineering  and  technical  information 
for  the  whole  inventory  of  weapons  systems,  as  well  as 
replacement  and  spare  parts.  (Green,  1992:53) 

Two  benefits  of  CALS  are  to  reduce  the  amount  of  paper 

used  and  derive  major  cost  savings  through  increased 

efficiency  through  standards  which  enable  data  exchange.  A 

Ticonderga-class  Navy  cruiser  carries  over  26  tons  of 

technical  data.  This  load  would  be  significantly  reduced  if 

the  technical  data  are  in  electronic  form  rather  than  in 

paper  form  (Green,  1992:53).  According  to  Green's  article, 

it  is  estimated  that  over  30  percent  of  the  life  cycle  cost 

of  weapon  systems  is  related  to  documentation.  The  Air 

Force  alone  spent  $7.5  billion  in  1989  on  technical 

information.  With  CALS,  DOD  estimates  that  a  savings  of 

over  40  percent  of  the  current  acquisition  cost  could  be 

cut.  This  amounts  to  a  $10  billion  savings  a  year  (Green, 

1992:53) . 

The  PMA  eliminates  the  need  for  the  maintenance 
technician  to  transport  heavy  and  bulky  paper  manuals  to  the 
weapon  system  being  repaired  (Tyree,  1992:34).  And  for 
weapon  systems  with  technical  order  (TO)  libraries  on  board, 
such  as  the  C-17,  the  paper-based  manuals  will  be  replaced 
with  computer  mass-memory  cartridges  used  with  the  PMA 
(Masquelier,  1992).  The  computer  presents  to  the  technician 
all  the  technical  data  needed  to  troubleshoot  and  repair  the 
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weapon  system  and  includes  the  ability  to  interface  with  the 
weapon  system  and  help  diagnose  the  problem. 

Rather  than  a  page  orientation  (format  used  for  paper- 
based  technical  manuals),  the  data  are  stored  in  an 
interactive  database  which  allows  the  technician  to  see  only 
the  applicable  data  and  gives  the  technician  the  ability  to 
branch  out  in  the  PMA  software  to  associated  areas.  For 
example,  during  the  troubleshooting  of  the  weapon  system, 
the  technician  discovers  that  part  A  is  bad.  In  the  page- 
based  system,  the  technician  would  have  to  look  in  another 
technical  manual  to  find  the  part  number.  With  the 
interactive  system,  the  technician  presses  a  key  to  find  the 
part  number.  The  concept  is  to  integrate  all  the 
information  required  for  flightline  maintenance  into  a 
single,  easy  to  use  source  for  the  technician  (Caporlette, 
1992:51).  As  stated  earlier,  the  prototype  PMA,  developed 
by  AL  personnel,  incorporates  a  graphical  user  interface. 

GUIs .  Graphical  user  interfaces  are  becoming  more 
popular  as  demonstrated  by  the  popularity  of  Macintosh* 
computers  and  the  Microsoft  Windows*  software  program  for 
the  personal  computer  (Morrissey,  1992:133).  Possible 
reasons  behind  such  growing  popularity  is  that  "the  use  of 
graphical  interfaces  systems  has  potential  for  increasing 
user  satisfaction,  reducing  training  time,  and  improving 
accuracy"  (Morris,  1990:2). 
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A  Study  by  James  Hollan  and  others  lists  important 
cognitive  properties  of  GUIs: 


1.  They  provide  a  physical  representational  system 
which  allows  users  to  better  understand  eUss tract 
relationships  and  make  use  of  the  brain's 
powerful  pattern-matching  ability. 

2.  Graphics-based  systems  make  the  depiction  of  models 
possible  that  are  similar  to  the  mental  models  or 
simulations  people  use  to  reason  about  the 

world. 

3.  Graphical  interfaces  better  depict  physical  state 
information  or  causal  connections  and  allow  the 
user  to  see  changes  in  the  state  of  the  system. 

4.  Graphical  systems  provide  the  potential  of  directly 
manipulable  representations  of  systems.  (Hollan 
and  others,  1986:25-26) 

Because  of  the  cognitive  properties  and  growing 
popularity  of  GUIs,  AL  personnel  decided  to  develop  a  GUI 
instead  of  a  text  based  interface  for  the  PMA.  The  PMA  is 
an  example  of  an  of  interactive  electronic  technical  manual 
or  lETM. 

lETMs .  An  interactive  electronic  technical  manual 

(lETM)  is  a  "package  of  information  needed  for  the  diagnosis 

and  maintenance  of  a  weapon  system,  optimally  arranged  and 

formatted  for  interactive  screen  presentation  to  the  end 

user  on  an  electronic  display  system"  (Department  of  Navy, 

1992:1).  The  technical  information  is  based  on  the 

electronic  form  rather  than  the  paper-based  form.  According 

to  Rainey  and  Fuller,  there  are: 

Deleterious  effects  of  page-oriented  information 
presentation  on  user  performance ... in  terms  of 
difficulty  in  finding  the  required  information,  in 
frequently  poor  comprehensibility,  and  in  the  presence 
of  technical  errors  which  were  not  eliminated  during 
the  preparation  and  inspection  procedures.  (Rainey  and 
Fuller,  1991:10) 


These  effects  could  lead  to  maintenance  errors, 
increased  time-to-repair,  false  part  removals,  and  increased 
training  time.  According  to  Chelouche,  problems  with  paper- 
based  technical  manuals  (TMs)  are  nvunerous.  Some  of  the 
problems  vdth  paper-based  manuals  are: 

1.  Weight  and  space  demands  for  TM  libraries. 

2.  Time  required  to  locate  data  within  a  given  manual. 

3.  Requirements  to  reference  other  technical  manual, 
resulting  in  a  mass  of  paper  required  to  perform 
maintenance  tasks. 

4.  Low  quality  updates  due  to  reliance  on  manual 
insertion  of  new  correction  pages  or  pen-and-ink 
correction  made  to  existing  pages. 

5.  High  investment  required  for  training  technicians. 

6.  Rising  costs  associated  with  maintaining  TMs. 

7 .  Low  quality  maintenance  due  to  the  discrepancy 
between  the  reading  capability  of  the  technician 
and  the  language  and  format  of  the  manuals. 

8.  Lack  of  guidance  from  experienced  personnel, 
thereby  resulting  in  a  higher  rate  of  maintenance 
errors  and  increased  down-time.  (Chelouche, 

1992:17) 

lETMs,  such  as  the  PMA,  were  designed  to  eliminate  or 
lessen  the  above  problems. 


Specific  Problem 

Armstrong  Laboratory  engineers  used  their  experience 
and  research  to  develop  the  prototype  GUI;  however,  they 
never  fully  tested  the  usefulness  of  access/navigational 
features  of  the  prototype  GUI  installed  on  the  prototype 
PMA.  This  lack  of  testing  leads  to  the  research  question: 
Which  access/navigational  features  enable  the  user  to  access 
information  from  the  PMA  with  the  highest  degree  of  user 
satisfaction? 
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For  the  purpose  of  this  research  the  following 
definitions  are  used: 

User  Satisfaction.  User  satisfaction  is  a  subjective 
rating  determined  by  the  user  based  on  the  fulfillment  of 
the  user's  expectations. 

Dedicated/Hardware  Keys .  Dedicated/Hardware  keys  keep 
the  same  function  regardless  of  the  current  screen  being 
displayed.  Examples  of  dedicated  keys  are  the  NEXT  and 
CANCEL  buttons  ( See  Figure  1 ) . 

Programmable  Soft  Kevs.  Programmable  soft  keys  are 
unlabeled  keys  whose  functions  are  determined  by  the  current 
screen.  Although  these  keys  are  hardware  components,  their 
function  changes  from  screen  to  screen  (See  Figure  1). 


Figure  1.  Location  of  Key  Components  on  the  PMA 
(Eagle  Technology,  1991:3) 


Push  Button  Keys.  Push  button  keys  are  similar  to 
programmable  soft  keys  in  that  their  functionality  changes 
depending  on  the  screen;  however,  the  button  is  presented  on 
the  screen  and  is  accessed  by  an  on-screen  move  and  select 
action,  not  a  hardware  key  action. 

Menu  Bar.  The  menu  bar  is  the  area  at  the  top  of  the 
display  device  screen  used  to  access  available  system 
functions.  Through  the  menu  bar  the  user  can  also  access 
pull-down  menus  or  submenus. 

Cursor  Control  Keys.  Cursor  control  keys  are  the  four 
directional  arrow  hardware  keys  used  to  access  or  navigate 
the  screen  (See  Figure  1). 

Number  Keys.  Number  keys  are  dedicated  keys  that  are 
used  to  input  numbers  or  access  the  menu  bar  (See  Figure  1). 

Research  Objectives 

The  objective  is  to  determine  which  access/navigational 
features  enable  the  user  to  access  information  from  the  PMA 
with  the  highest  degree  of  user  satisfaction.  The 
experiment  focuses  on  six  access/navigational  features 
installed  on  the  prototype  GUI.  The  experiment  is  divided 
into  two  main  tasks.  The  first  task  evaluates  screen  access 
and  navigation  using  the  following  three  features: 
dedicated/hardware  keys,  programmeUsle  soft  keys,  and  push 
button  keys.  The  second  task  evaluates  menu  access  and 
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navigation  using  the  following  three  features:  number  keys, 
cursor  control  keys,  and  programmable  soft  keys. 

By  finding  the  best  navigational/access  features, 
redundant  features  can  be  eliminated,  and  software  memory 
requirements  reduced.  Eliminating  the  options  available  on 
the  GUI  will  ultimately  save  the  DoD  money  and  make  the 
system  easier  for  the  technician  to  learn  and  use. 

Investigative  Questions 

To  answer  the  research  question,  the  following  areas 
are  investigated: 

1.  What  are  the  differences  in  total  task  completion 
times  eunong  the  three  features  in  each  main  task? 

2.  What  are  the  differences  in  total  key  strokes  among 
the  three  features  in  each  main  task? 

3.  What  is  the  correlation  between  total  task 
completion  times  and  total  key  strokes? 

4.  What  is  the  correlation  between  total  task 
completion  times  and  user  satisfaction? 

5.  What  is  the  correlation  between  total  key  strokes 
and  user  satisfaction? 

6.  Which  access/navigational  feature  provides  the  most 
user  satisfaction? 

7.  What  is  the  statistical  difference  between  non- 
experienced  and  experienced  computer  users  when  rating  the 
features  according  to  user  satisfaction? 
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The  scope  of  this  research  was  limited  to  the  prototype 
GUI  on  the  prototype  PMA  developed  by  AL  personnel.  The  lab 
personnel  developed  the  GUI  using  the  military  specification 
MIL-M-87268,  hereby  referred  to  as  the  lETM-GCSFUI,  which 
gives  guidelines  for  designing  GUIs.  The  specification 
states : 

A  common  set  of  user-interface  components  and 
presentation  conventions  shall  be  used  to  provide  a 
consistent  user-interface  across  all  presentations 
devices.  These  user-interface  components  and 
presentation  conventions  are  common  to  most  Graphical 
User  Interfaces.  The  common  interface  components 
described  in  this  specification  shall  be  implemented  on 
all  of  the  various  types  of  presentation  systems  from  a 
large  screen  device  to  a  portable  small  screen  device. 
(Department  of  Defense,  1992:21) 

The  screens  designed  for  the  experiment  followed  the 
guidelines  outlined  in  the  lETM-GCSFUI.  Specifically,  the 
sections  dealing  with  the  following:  common  user  interface 
components,  cursor  controls,  menu  systems,  display 
formatting,  general  display  format  for  lETMs  and  required 
navigation  functions.  These  areas  are  located  in  sections 
3.4.1  to  3.5.2  in  the  lETM-GCSFUI  (Department  of  Defense, 
1992:21-43) . 

In  order  to  keep  the  experiment  focused  on  comparing 
access/navigational  features,  we  designed  an  experiment 
using  screens  that  specifically  evaluate  the  features  being 
tested.  The  screens  contained  maintenance  information,  but 
were  limited  to  simple  operations  to  keep  the  focus  on 
testing  the  access/navigational  features  and  not  the 


technical  data.  Because  of  the  modified  computer  screens,  a 
personal  laptop  computer  was  used  to  simulate  a  PMA.  The 
keys  on  the  PMA  and  personal  computer  keyboard  are  not 
exactly  the  same.  This  dissimilarity  was  a  limitation,  but 
should  not  have  adversely  affected  the  results. 

The  experiment  was  conducted  in  a  controlled 
environment  inside  a  maintenance  avionics  shop. 

Additionally,  the  study  was  limited  to  Air  Force  maintenance 
personnel  who  were  stationed  locally  at  Wright-Pat ter son 
AFB,  OH. 

SvHwnftry 

Portable  maintenance  aids  help  the  technician  maintain, 
diagnose,  and  repair  malfunctions  on  a  weapon  system.  Two 
studies  have  demonstrated  the  need  for  PMAs.  These  studies 
validated  the  usefulnest^  of  such  aids  in  both  the  in-shop 
and  flight line  environment.  Another  study  compared  input 
devices,  joystick  and  four-cursor  keys,  on  the  PMA.  The 
results  of  this  study  suggested  that  there  were  no 
significant  statistical  differences  between  the  input 
devices.  Another  study  compared  GUIs  with  text  and  command 
based  interfaces,  stating  that  GUIs  decreased  training  time 
and  provided  a  consistent  interface.  There  is,  however,  a 
need  to  evaluate  access/navigational  features  of  the 
prototype  GUI  installed  on  the  prototype  PMA. 
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In  this  chapter,  we  discussed  CALS,  GUIs,  and  lETMs. 
Additionally,  we  defined  the  specific  problem  to  be 
investigated:  Which  access /navigation  features  enable  the 
user  to  access  information  from  the  PMA  with  the  highest 
degree  of  user  satisfaction?  We  also  stated  the 
investigative  questions,  and  discussed  the  scope  and 
limitations  of  the  research. 

Chapter  II,  Literature  Review,  contains  background 
information,  and  further  establishes  the  need  for  research. 
Chapter  III,  Methodology,  describes  the  details  of  the 
experiment.  Chapter  IV,  Findings  and  Analysis,  provides  the 
results  of  the  experiment,  more  specifically,  it  answers  the 
investigative  questions.  Chapter  V,  Conclusion  and 
Recommendations,  lists  conclusions  and  recommendations  for 
follow-on  experiments.  A  list  of  acronyms  used  in  this 
thesis  is  included  in  Appendix  A. 
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The  weapon  systems  of  the  Department  of  Defense  (DoD) 
are  becoming  increasingly  complex.  This  complexity  of 
design  directly  influences  the  complexity  of  maintenance 
performed  on  the  weapon  system.  Interactive  electronic 
technical  data  (versus  paper-based  technical  data)  are 
needed  to  maintain  advanced  systems,  such  as  the  B-2  and  the 
F-22.  Automation  promotes  speed  and  accuracy  in  updating 
data,  and  allows  technicians  to  have  the  most  current 
diagnostic  and  technical  information  at  their  finger  tips. 
More  importantly,  the  information  is  integrated  and 
interactive,  which  results  in  faster  accessing  times,  fewer 
false  part  removals,  increased  productivity  of  inexperienced 
technicians,  and  reduced  maintenance  downtimes. 

One  way  for  the  technician  to  access  this  information 
is  through  a  portable  maintenance  aid  (PMA),  a  computer  that 
is  easily  transported  to  the  flightline.  The  prototype  PMA 
incorporates  a  graphical  user  interface  (GUI)  versus  a  text/ 
command-based  interface.  The  Armstrong  Laboratory  tested 
the  feasibility  of  a  PMA,  but  never  fully  tested  the 
usefulness  of  certain  access /navigational  features  of  the 
prototype  GUI .  Our  research  evaluates  certain 
access/navigational  features  of  the  GUI  to  determine  the 


features  with  the  highest  degree  of  user  satisfaction.  The 
purpose  of  our  experiment  is  to  identify  which 
access/navigational  feature  (specified  in  HIL-M-87268)  work 
best  given  the  maintenance  data  and  environment.  This 
effort  is  to  ensure  that  access/navigational  features  of 
prototype  maintenance  PMA  GUIs  are  suitable  for  future  and 
emerging  progreuns  such  as  the  B-2  and  F-22  programs. 

The  purpose  of  this  chapter  is  to  provide  the  framework 
for  our  research  on  testing  certain  access/navigational 
features  of  the  prototype  GUI.  First,  we  review  the 
evolution  of  automated  technical  data  and  diagnostic 
systems,  beginning  with  the  Computer-aided  Acquisition  and 
Logistic  Support  (CALS),  and  automated  technical  orders.  We 
then  discuss  the  Computer-based  Maintenance  Aids  System 
(CMAS)  and  the  Integrated  Maintenance  Information  System 
(IMIS).  We  conclude  with  past  research  on  PMAs  and  GUIs, 
and  establish  the  need  for  research. 

Computer-aided  Acquisition  and  Logistic  Support  fCALS) 

CALS  resulted  from  the  DoO  effort  to  automate  and 
streamline  the  purchase,  maintenance,  modification,  and 
documentation  of  activities  conducted  to  support  a  weapon 
system.  The  first  objective  of  CALS  is  to  automate  the 
delivery  of  technical  information,  including  engineering 
drawings  and  technical /logistics  support  data,  from  the 
contractors  to  the  users.  The  second  objective  of  CALS  is 
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to  create  a  unified  data  base  for  information  sharing  among 
DoD  and  industry  (O'Neal,  1989:42). 

A  DoD  task  force  was  formed  in  1984  to  study  the 
feasibility  of  a  DoD-wide  computer-aided  logistic  support 
system.  The  task  force  was  composed  of  senior  industry  and 
government  personnel  with  the  official  charter  to  develop  a 
strategy  and  recommend  a  master  plan  for  CALS 
implementation.  The  task  force  cited  the  following 
logistics  problems  that  CALS  could  alleviate: 

1.  Incomplete/illegible  engineering  drawings. 

2.  Inaccurate/incomplete  re-procurement  documentation. 

3.  Outdated  technical  data. 

4.  Expensive/enormous  paper-based  technical 
information . 

5.  Inadequately  documented  configuration  data. 
(Correale,  1987:192) 

In  August  1988  the  Deputy  Secretary  of  Defense  William 
Taft  issued  a  memorandum  instructing  program  managers  to 
include  CALS  standards  in  new  weapon  system  acquisitions 
(Taft,  1988:1).  This  memo  was  important  to  the  CALS  program 
because  it  gave  CALS  renewed  interest  at  the  highest  levels. 

Two  lead  programs  have  driven  the  CALS  effort:  the 
Navy's  Engineering  Data  Management  Information  and  Control 
System  (EDMICS),  awarded  to  PRC  Inc  in  1989,  and  the  Army's 
program.  Army  CALS  (ACALS),  awarded  to  Computer  Science  Corp 
in  Fall  1991  (Jenks,  1992:28).  In  1991,  ACALS  was  renamed 
J-CALS  after  combining  with  the  Air  Force's  CALS  program — 
Joint  Uniformed  Service  and  Technical  Information  System 
(JUSTIS).  In  1992,  much  of  the  momentum  of  CALS  shifted 
from  EDMICS  to  J-CALS  (Jenks,  1992:28). 
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EOMICS  is  concerned  with  converting  engineering 
drawings  from  huge  stacks  of  paper  to  electronic  digits. 
J-CALS/  however,  is  more  of  a  day-one  forward  system 
designed  to  support  the  weapon  system  throughout  its  life 
cycle,  including  all  associated  paperwork.  As  stated 
earlier,  the  first  objective  of  CALS  is  to  automate  and 
standardize  the  delivery  of  technical  information.  Accurate 
technical  orders  are  needed  to  maintain  and  support  today's 
weapon  systems. 

Technical  Orders 

Technical  Orders  (TOs)  are  used  by  Air  Force 
technicians  to  help  maintain  and  support  weapon  systems. 

The  current  paper-based  system  is  extremely  bulky. 

According  to  Major  McClain  of  the  P-16  program  office,  the 
F-16  fighter  aircraft  reguires  1,963  TOs  to  maintain  and 
operate  it  (McClain,  1993).  It  is  estimated  that  one  B-1 
bomber  alone  reguires  over  1.5  million  technical  drawings  to 
maintain  and  support  it  (Jenks,  1992:37).  Not  only  is  the 
current  system  paper  intensive,  but  keeping  the  manuals 
updated  involves  many  manhours. 

If  the  technical  data  change,  manuals  have  to  be 
updated  using  the  original  paper  source.  The  Navy  reguires 
approximately  1,000  sailor-years  annually  just  to  keep  its 
manuals  current  (Nordwall,  1990:66).  According  to  Kerr,  at 
any  one  time,  10  percent  of  the  information  in  TOs  is 
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outdated  (Kerr,  1988:81).  Outdated  technical  information 
leads  to  incorrect  maintenance  actions,  which  can  result  in 
loss  of  life,  safety  problems,  or  damage  to  expensive  parts. 
Technicians  use  computers  not  only  to  access  automated 
technical  information,  but  also  to  access  diagnostic  aids 
used  to  troubleshoot  weapon  system  malfunctions. 

Computer-based  Maintenance  Aids  System  (CMAS) 

The  Computer-based  Maintenance  Aids  System  (CMAS) 
development  program  introduced  the  use  of  computers  to 
present  maintenance  procedures  for  intermediate-level 
equipment.  According  to  Donald  Thomas  and  Jeffrey  Clay,  the 
initial  attempt  began  in  1978  to  develop  an  automated 
technical  data  presentation  system  (ATDPS)  for  use  in 
intermediate  level  maintenance  shops  (off -equipment 
maintenance).  The  initial  focus  of  this  effort  was  on  the 
development  of  human  factors  and  technical  data 
requirements.  However,  a  CMAS  prototype  was  never  built 
because  of  an  operational  requirement  requiring  the  computer 
to  be  deployable  (Thomas  and  Clay,  1988:4). 

Thomas  and  Clay  state  that  in  1982,  AL  made  a  second 
attempt  to  develop  an  ATDPS  prototype  for  intermediate  level 
maintenance.  CMAS-I,  as  it  was  called,  used  commercially 
available  hardware  and  adapted  the  Navy  Technical 
Information  Presentation  System  for  its  use.  CMAS-I  was 
installed  in  an  intermediate-level  avionics  maintenance  shop 
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at  Offutt  AFB,  Nebraska,  for  evaluation.  However,  CMAS-I 
failed  to  validate  its  usefulness  because  the  program  lacked 
acceptance  from  the  users,  primarily  due  to  slow  response 
times  and  ineffective  man-machine  interface  techniques 
(Thomas  and  Clay,  1988:5). 

A  follow-up  to  CHAS-I,  called  CMAS-II,  used  a  Grid 
Compass  Model  1139  microcomputer  for  its  hardware  and 
incorporated  many  of  the  design  features  in  CMAS-I. 

However,  priority  was  placed  to  improve  the  response  times 
and  interactions  between  man  and  machine.  CMAS  II  was 
installed  at  Grissom  AFB,  Indiana,  in  June  1988.  After 
gaining  acceptance  from  the  users,  and  demonstrating  the 
viability  of  computer-based  maintenance  aids,  the  CMAS-II 
was  tested  again  by  the  Navy.  Following  these  two 
successful  tests,  specifications  were  developed  for 
automated  technical  data  systems  (hardware)  and  for 
technical  data  presentation  (software)  (Thomas  and  Clay, 
1988:6).  Although  the  CMAS  project  ended,  work  based  on  the 
CMAS  results  continues  under  the  Integrated  Maintenance 
Information  System  program. 

Integrated  Maintenance  Information  System  (IMIS) 

The  Armstrong  Laboratory  has  been  testing  and 
developing  the  Integrated  Maintenance  Information  System 
(IMIS)  since  the  termination  of  CMAS.  IMIS  was  developed  to 
access  and  integrate  maintenance  information  from  multiple 
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sources  and  present  the  information  to  technicians  through  a 
single  integrated  source.  On  the  flightline,  the  single 
source  is  the  hand-held  computer,  the  PMA.  IMIS  was 
developed  in  an  attempt  to  consolidate  other  maintenance 
computer-based  information  systems  (Department  of  the  Air 
Force,  1987:1).  Current  computer-based  information  systems 
include  the  Comprehensive  Engine  Management  System  (CEMS- 
IV),  and  the  Core  Automated  Maintenance  System  (CAMS).  One 
benefit  of  the  IMIS  is  that  technicians  will  only  have  to 
learn  one  system  but  will  have  information  available  from 
many  different  logistic  databases. 

IMIS  is  a  comprehensive  maintenance  system  that  links 
the  flightline  with  the  maintenance  information  workstation 
(MIW)  (See  Figure  2). 


(Masquelier,  1992) 
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IMIS  is  divided  into  hardware  and  software  subsystems. 
The  following  section  discusses  the  siibsystems  in  more 
detail  (Department  of  the  Air  Force,  1987:45-47). 

Portable  Maintenance  Aid  (PMA).  The  PMA  is  a 
compact,  lightweight,  battery-powered,  portable  computer, 
rugged  enough  for  flight line  use.  The  PMA  displays 
technical  order  information,  which  is  stored  on  removable 
memory  cartridges.  Besides  displaying  technical 
information,  the  PMA  has  diagnostic  aids  installed  to  help 
the  technician  troubleshoot  difficult  maintenance  problems. 

Maintenance  Information  Workstation  (MIV) .  The  MIW  is 
an  in-shop  workstation  with  an  interface  computer.  The  MIW 
can  interface  with  the  PMA  and  other  maintenance  information 
systems,  like  CAMS  and  CEMS-IV. 

Integration  Software.  Integration  software  allows  the 
technician  to  access  all  the  information  through  one  device. 
Before  IMIS,  technicians  had  to  learn  several  different 
maintenance  information  systems  to  accomplish  the  task. 

Applications  Software.  Applications  software  is  a 
comprehensive  database  used  to  troubleshoot  and  maintain  the 
weapon  system.  This  database  incorporates  various  sources 
of  information  in  one  place. 

As  the  IMIS  concept  evolved,  AL  personnel  designed 
several  prototype  PMAs  which  incorporated  the  use  of  GUIs. 
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Portable  Maintenance  Aids 


Accurate,  up-to-date  information  is  needed  to  maintain 
today's  complex  weapon  systems.  How  the  technicians  receive 
this  information  has  been  the  focus  of  past  research.  In 
1991,  Barbara  Masquelier  researched  the  feasibility  of  a 
head  mounted  display  device  (HMDD)  because  a  flat  screen 
computer  has  limitations.  A  flat  screen  computer  does  not 
physically  fit  in  all  compartments  of  an  aircraft, 
prohibiting  technicians  from  accessing  necessary  information 
at  all  times  (Thomas  and  Clay,  1990).  To  help  alleviate 
these  limitations,  Masquelier  compared  the  monocular  HMDD  to 
a  portable  laptop  computer,  using  avionics  technicians  in  an 
in-shop  repair  environment.  Her  research  objective  was  to 
determine  if  a  performance  difference  existed  on  the  basis 
of  the  display  device  (HMDD  vs  laptop  computer).  She  found 
no  statistically  significant  evidence  of  one  device 
improving  performance  over  the  other  (Masquelier,  1991:67). 

A  follow-on  study  to  Masquelier 's  research  was 
conducted  in  1992  by  Jeffrey  Friend  and  Randy  Grinstead. 

They  expanded  Masquelier 's  research  to  a  flightline 
environment.  They  conducted  two  maintenance  tasks  to 
determine  the  effects  of  the  display  type  (HMDD  or  PMA)  on 
the  technician  performance.  Their  study  concluded,  "...that 
there  are  some  tasks,  specifically  complex  procedural  tasks 
and  inspection  tasks,  that  may  benefit  from  the  use  of  an 
HMDD"  (Friend  and  Grinstead,  1992:75). 
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Gerald  Streff  and  Robert  Gundel  also  conducted  research 
in  1992  on  the  PMA.  Their  research  focused  on  the 
comparison  of  four-cursor  buttons  and  a  joystick  to  access 
computerized  technical  information.  They  concluded:  "Data 
collected  during  this  experiment  indicated  that  there  are  no 
statistical  differences  in  access  times  using  either  the 
joystick  device  or  the  four-cursor  key  configuration" 

(Streff  and  Gundel,  1992:65).  AL  personnel  tested  different 
types  of  PMAs  and  hardware  input  devices,  but  never  tested 
certain  access/navigational  features  of  the  prototype  GUI. 

Graphical  User  Interfaces 

According  to  Ralph  Sabene,  the  GUI  has  become  the 
standard  operating  environment  for  most  software  programs  in 
the  last  ten  years  (Sabene,  1992:2).  The  Xerox  Palo  Alto 
Research  Center  introduced  the  first  GUI  in  the  1970s  (Baran 
and  Hayes,  1989:250). 

Apple  Computer  introduced  the  next  GUI  with  the  Lisa® 
computer  in  1983.  However,  it  wasn't  until  the  introduction 
of  the  Macintosh®  Computer  in  1984  that  the  GUI  became 
commercially  successful.  According  to  Sabene,  "most  people 
today  credit  the  Apple  Macintosh®  as  starting  the  'GUI 
revolution'"  (Sabene,  1992:2).  Later,  Microsoft 
Corporation  contributed  to  the  GUI  popularity  by  introducing 
a  GUI,  Windows®,  for  IBM  compatible  computers  (Sabene, 
1992:2) . 
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However,  Windows*  and  Macintosh*  are  not  the  only  GUIs 
available.  The  popularity  of  GUIs  is  widespread,  crossing 
many  hardware  and  software  platforms.  Table  1  shows  the 
available  hardware/software  platforms  for  GUIs. 


TABLE  1 

AVAILABLE  HARDWARE /SOFTWARE  PLATFORMS 
USING  GRAPHICAL  USER  INTERFACES 


1  Software/Hardware  Platforms 

GUIs  1 

Macintosh 

Microsoft 

Windows  3 . 1 

IBM 

OS/2  2.0 

Digital  Equipment  Corporation 

DECwindows 

Open  Software  Foundation 

Motif 

Santa  Cruz  Operation  (SCO) 

Open  Desktop 

Commodore  Amiga 

Intuition 

NeXT  Computer 

NeXT  Step 

Digital  Research 

GEM 

Sun  Microsystems 

Open  Look 

Hewlett-Packard  with  Microsoft 

Common  X  Interface  1 

1  Hewlett-Packard 

NewWave  | 

(Adapted  from  Baran  and  Hayes, 1989) 


GUI  Components.  GUIs  differ  from  text/command  based 
interfaces  by  their  manipulation  of  graphics  or  pictorial 
representations  to  achieve  a  command  or  function  rather  than 
through  a  command  that  must  be  typed.  Baran  and  Hayes 
identified  a  list  of  parts  that  are  usually  part  of  GUIs. 
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They  are  listed  below  with  a  short  description: 

1.  A  pointing  device,  usually  a  mouse  or  cursor  used 
to  select  items  in  the  interface.  Other  pointing 
devices  include  light  pens  and  joysticks. 

2.  Through  use  of  a  pointing  device,  menus  can  pop  up  when 
a  menu  item  is  selected. 

3.  Windows  that  graphically  display  what  the  computer 
is  doing. 

4.  Icons  that  represent  files,  directories,  etc. 

5.  Dialogue  boxes,  buttons,  sliders,  check  boxes,  and 
a  plethora  of  other  graphical  widgets  that  let 
you  tell  the  computer  what  to  do  and  how  to  do 

it.  (Baran  and  Hayes,  1989:250) 

A  GUI  does  not  have  to  include  all  these  features  in 
order  to  be  considered  a  GUI;  however,  most  GUIs  include 
many  of  these  features. 

Graphical  User  Interface  Design.  In  order  to  design  a 
GUI,  the  designer  can  follow  guidelines  established  by  many 
researchers  in  the  field  of  interface  design  and  human- 
computer  interface  design.  A  search  of  the  past  research 
shows  that  Sidney  Smith  and  Jane  Hosier  identified  944 
guidelines  for  designing  user  interface  software  (Smith  and 
Hosier,  1986).  The  guidelines  range  from  the  general  to 
specific  details. 

In  contrast,  Ben  Schneiderman  identifies  eight  general 
guidelines  which  "provide  a  good  checklist  for  ensuring  that 
a  design  has  at  least  included  consideration  of  issues 
commonly  applicable  to  user  interfaces"  (Peabody,  1991:7). 
Schneiderman 's  eight  principles  are  listed  in  Table  2. 
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TABLE  2 


SCHNEIDERMAN'S  EIGHT  USER- INTERFACE  PRINCIPLES 


1. 

Strive  for  consistency 

2. 

Enable  frequent  users  to  use  shortcuts 

3. 

Offer  informative  feedback 

4. 

Design  dialogues  to  yield  closure 

5. 

Offer  simple  error  handling 

6. 

Permit  easy  reversal  of  actions 

7. 

Support  internal  locus  of  control 

8. 

Reduce  short-term  memory  load 

(Adapted  from  Schneiderman, 1987 ) 


Schneiderman ' s  first  principle,  consistency,  is 
important  because  it  allows  the  user  to  focus  on  the  task 
rather  than  concentrating  on  how  to  maneuver  around  the 
interface.  Under  the  IMIS  concept,  maintainers  will  use 
several  different  computers  to  accomplish  their  job.  Having 
a  consistent  interface  on  both  the  PMA  and  the  in-shop 
workstation  decreases  training  time  because  only  one 
interface  has  to  be  learned.  When  designing  a  good 
interface,  the  designer  should  encompass  as  many  as  the 
Schneiderman ' s  eight  principles  as  possible. 

The  Need  for  Research 

According  to  Barbara  Masguelier,  Human  Factors 
Engineer,  AL  personnel  performed  field  evaluations  using  the 
PMA  and  the  prototype  GUI,  but  never  specifically  compared 
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access/navigational  features  of  the  prototype  GUI. 

Laboratory  personnel  feel  it  is  important  to  evaluate 
access/navigational  features  of  the  GUI  as  outlined  in  the 
military  specification  {MIL-M-872_68 )  governing  GUIs. 

Through  formal  evaluation  of  GUI  features,  assessments  can 
be  made  as  to  which  access/navigational  features  accommodate 
good  user  interface  design  for  maintenance  data  and 
environments.  Our  research  evaluates  certain 
access /navigational  features  of  the  GUI  to  determine  which 
features  provide  the  highest  degree  of  user  satisfaction. 

Summary 

In  this  chapter,  we  discussed  the  evolution  of  the 
portable  maintenance  aid  and  the  graphical  user  interface. 

We  discussed  the  Computer-aided  Acquisition  and  Logistic 
Support  (CALS)  initiative,  automated  technical  orders,  and 
the  first  test  of  a  PMA — Computer-based  Maintenance  Aids 
System  (CMAS).  We  reviewed  the  Integrated  Maintenance 
Information  System  (IMIS)  program  and  past  research  on  PMAs. 
We  concluded  with  past  research  on  GUIs,  and  established  the 
need  for  research.  Chapter  3,  Methodology,  outlines  the 
details  of  the  experiment,  including  the  experimental  design 
and  method  of  analysis. 
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Introduction 

As  stated  in  Chapter  1,  portable  maintenance  aids 
(PMAs)  are  needed  to  support  and  maintain  weapon  systems.  A 
graphical  user  interface  (GUI)  is  installed  on  the  PMA;  the 
current  prototype  GUI,  the  AL  demonstration  device,  has 
redundant  access/navigational  features.  The  purpose  of  this 
research  is  to  determine  the  "best"  access/navigational 
feature  to  use  on  future  PMAs  and  to  evaluate  those  features 
specified  in  the  IETM-GC&.  1.  Best  feature  is  defined  as 
the  feature  providing  the  highest  mean  value  for  overall 
user  satisfaction.  To  find  the  best  feature,  we  conducted 
an  experiment  evaluating  six  access/navigational  features. 
First  we  discuss  the  experimental  design.  Then  we  address 
the  elements  of  the  experiment,  including  subjects,  tasks, 
training,  and  hardware/software.  Next,  we  discuss  data 
collection  and  review  the  investigative  questions.  We  then 
address  the  method  of  analysis  and  conclude  with  a  brief 
summary  of  the  chapter. 

Exp^riPbhthi  P^gjgn 

Researchers  use  experiments  to  gather  information  to 
test  hypotheses.  Experiments  have  many  advantages  over 
other  methods  in  collecting  data.  According  to  Emory  and 
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Cooper,  "the  foremost  advantage  is  the  researcher's  ability 
to  manipulate  the  independent  variables"  (Emory  and  Cooper, 
1991:418).  This  allows  the  researcher  to  determine  if 
changes  in  the  dependent  variable  are  a  result  of  changes  in 
the  independent  variable. 

Another  advantage  of  experiments  is  that  the  researcher 
has  greater  control  over  contamination  from  extraneous 
variables.  This  helps  the  researcher  "isolate  experimental 
variables  and  evaluate  their  impact  over  time*  (Emory  and 
Cooper,  1991:418).  Additionally,  experiments  lead  to  "an 
average  effect  of  the  independent  variable  across  people, 
situations,  and  times*  because  experiments  can  be  repeated 
or  replicated  with  different  subject  groups  and  conditions 
(Emory  and  Cooper,  1991:418). 

However,  the  artificiality  of  the  laboratory  is  one  of 
the  biggest  disadvantages  of  experiments  (Emory  and  Cooper, 
1991:418).  Subjects  may  react  differently  under 
experimental  settings  than  they  would  under  normal 
conditions.  Regardless,  experiments  are  still  one  of  the 
best  methods  in  determining  changes  in  dependent  variables 
caused  by  changes  in  the  independent  variable. 

This  research  incorporates  a  blocking  factor  for 
grouping  people  with  and  without  computer  experience. 
Randomized  block  experimental  designs  are  effective  when  the 
researchers  are  interested  in  learning  if  there  are 
differences  in  results  among  various  groups  of  subjects. 
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There  are  many  elements  of  designed  experiments. 

McClave  and  Benson  provide  the  following  definitions  of  the 
elements  of  designed  experiments: 

1.  Response/Dependent  Variable:  Variable  of  interest. 

2.  Factors/ Independent  Variaibles:  Varied>les  whose 
effect  on  the  response  is  of  interest. 

3.  Levels  of  Independent  Varied>les:  Values  of  the 
factors  utilized. 

4.  Treatments:  Factor-level  combinations  utilized. 
(McClave  and  Benson,  1991:860-861) 


The  response  of  interest  in  this  experiment  is  the  user 
satisfaction  level  for  the  different  access /navigational 
features  (factors)  used  for  the  two  main  tasks.  The  levels 
for  Task  1  are  dedicated/hardware  keys,  programmable  soft 
keys,  and  push  buttons  keys.  The  levels  for  Task  2  are 
number  keys,  cursor  keys,  and  programmable  soft  keys.  The 
treatments  are  the  order  in  which  the  access/navigational 
features  are  tested. 


Subjects 

Minimum  sample  size  was  determined  by  using  a  formula 
from  Diamond  (Diamond,  1981:27-33).  According  to  Diamond, 
"The  correct  sample  size  assures  the  experimenter  that  the 
risks  of  error  will  be  equal  to  or  less  than  a  and  §  when 
the  experiment  is  completed  (Diamond,  1981:33).  The  formula 
for  computing  sample  size  N  is: 


N  =  2(U.  -I-  0^/6^ 


(1) 
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where  U.  and  U,  are  the  normal  distribution  nurobers  for 
the  alpha  and  beta  risks,  (sigma-squared)  is  the  variance 
and  6  (delta)  is  the  engineering  factor.  The  alpha  risk  is 
the  probability  of  rejecting  the  null  hypothesis  when  it  is 
true.  The  beta  risk  is  the  probedaility  of  failing  to  reject 
the  null  hypothesis  when  it  is  false.  The  larger  the  sample 
size  N,  the  smaller  the  risk  of  an  alpha  or  beta  error. 

This  formula  calculated  the  minimum  sample  size  by 
using  alpha  and  beta  errors  equal  to  the  overall 
significance  level  (0.05  for  this  experiment)  and  a  delta 
equal  to  one  standard  deviation.  Using  Table  1  from 
Diamond's  book,  U,  =  1.960  and  =  1.282.  Using  these 
values  and  equation  1/  the  minimum  sample  size/  N,  is  21 
(Diamond/  1981:28-30/325).  A  total  of  28  maintenance 
technicians  from  the  Communication/  Navigation/  Guidance 
Control/  and  Electro-Environmental  shops  within  the  4950th 
Test  Wing  located  at  Wright-Patterson  AFB/  OH  and  avionics 
technicians  from  the  907th  Logistics  Support  Squadron/  also 
located  at  Wright-Patterson  AFB/  participated  in  this 
experiment.  The  4950th  Test  Wing  and  Reserve  personnel  were 
selected  mainly  because  of  their  availability/  but  also 
because  they  provided  a  good  cross-sectional  sample  of  the 
population  of  all  Air  Force  technicians. 

These  technicians  were  randomly  divided  into  two 
groups.  The  first  group  accomplished  Task  1  first  and  then 
Task  2.  The  second  group  accomplished  Task  2  first  then 
Task  1.  This  minimized  recency  effects.  Each  group  had 
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non-experienced  and  experienced  computer  users.  We  used 
background  surveys  (See  Appendix  B)  to  gather  background 
information  on  the  technicians.  We  distributed  the  survey 
to  the  various  shops.  The  surveys  yielded  13  non- 
experienced  and  15  experienced  computer  users  for  a  total  of 
28  subjects.  A  non-experienced  computer  user  is 
operationally  defined  as  having  less  than  three  years 
computer  experience,  and  an  experienced  computer  user  is 
defined  as  having  over  three  years  computer  experience. 

For  the  purpose  our  research,  maintenance  skill-level 
is  not  a  factor.  Since  we  are  evaluating  a  GUI  access/ 
navigational  feature,  maintenance  skill-level  should  not 
impact  the  experiment.  The  computer  screens,  however, 
contain  maintenance  information  from  the  F/A-18  technical 
manuals.  This  information  is  not  difficult  to  understand 
and  should  not  influence  the  test.  We  decided  to  design  the 
screens  with  maintenance  information  because  the  portable 
maintenance  aid  (PMA)  is  slated  for  the  maintenance  complex. 
The  maintenance  information  on  the  screens  is  generic,  and 
is  mainly  there  to  provide  a  realistic  environment  for  the 
experiment . 

TagKg 


To  evaluate  GUI  access/navigational  features,  we 
divided  our  experiment  into  two  main  tasks:  Screen 
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Access/Navigation  and  Menu  Access/Navigation.  Each  main 
task  evaluates  three  access /navigational  features: 


TASK  1:  Screen  Access/Navigation 

la.  Dedicated/Hardware  Keys 

lb.  Programmable  Soft  Keys 

lc.  Push  Button  Keys 
TASK  2:  Menu  Access /Navigation 

2a.  Number  Keys 
2b .  Cursor  Keys 
2c.  Programmable  Soft  Keys 
Tedsle  3  summarizes  the  experiment. 


TABLE  3 

GROUP  AND  TASK  ORDER  ASSIGNMENT 


liiOcifif 

I 

14 

Screen  Access /Navigation 

Menu  Access/Navigation 

II 

14 

Menu  Access/Navigation 

Screen  Access/Navigation 

Group  I  had  7  non-experienced  and  7  experienced  and 
Group  II  had  6  non-experienced  and  8  experienced.  Each  main 
task  consists  of  three  subtasks,  resulting  in  six  different 
combinations  for  subtask  assignment  order.  The  subjects 
were  randomly  assigned  to  one  of  the  six  subtask 
combinations  in  each  main  task  to  ensure  that  each  subtask 
combination  was  used  by  an  equal  number  of  subjects  to 
minimize  bias  results  due  to  recency  factors.  For  example. 
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Subject  1  would  perform  the  task  in  the  order  la,  lb,  Ic; 
whereas  Subject  2  would  perform  the  task  in  the  order  of  lb, 
la,  Ic  etc. 

The  subjects  were  required  to  access  information  and 
navigate  their  way  around  the  screen  using  one  access/ 
navigational  feature  until  all  three  subtasks  were 
completed. 

The  tasks  used  in  this  experiment  were  designed  by  the 
researchers  to  test,  one-at-a-time,  an  access/navigational 
feature.  To  accomplish  this,  we  designed  each  screen  so  all 
three  access/navigational  feature  could  be  used.  However, 
depending  on  the  subtask,  only  one  access/navigation  feature 
was  operational — the  other  two  features  were  locked  out. 

This  ensure"  that  only  the  feature  being  tested  was  used. 

A  pilot  test  was  conducted  using  11  students  and 
instructors  from  the  Air  Force  Institute  of  Technology 
stationed  at  Wright-Patterson  AFB,  OH.  The  purpose  of  the 
pilot  study  was  to  refine  the  experiment.  The  variability 
of  the  results  from  the  pilot  test  was  used  to  verify  the 
sample  size.  The  standard  deviations  were  small  enough  to 
yield  significant  differences  among  the  means,  therefore  a 
sample  size  of  21  or  greater  was  sufficient. 

Training 

Prior  to  each  subtask,  the  subject  was  shown  the 
relevant  keys  needed  for  that  subtask  (See  Appendices  C,  D) . 
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Instructions  for  all  tasks  were  provided  on  the  screens, 
eliminating  the  need  for  paper  instructions.  Instructions 
were  also  given  to  concentrate  on  the  access /navigational 
feature  and  not  the  maintenance  information  presented  on  the 
screens . 


Hardware  and  Software 

The  hardware  used  for  this  experiment  was  a  Compaq 
Contura  386SL  laptop  running  under  DOS  6.0*.  The  laptop  was 
used  to  simulate  a  PMA  because  of  its  portability,  weight, 
and  small  screen  size.  The  laptop  uses  a  25MHz  Intel  chip, 
with  a  84Mb  hard  drive,  4Mb  of  RAM  and  a  monochrome  LCD 
screen.  During  the  experiment,  the  laptop  used  AC  power 
with  all  power  management  controls  turned  off.  Using  AC 
power  instead  of  battery  power  allowed  the  chip  to  run  100 
percent  of  the  time,  producing  more  consistent  results. 

The  screens  used  in  this  experiment  were  programmed 
using  Visual  Basic^”  2.0  for  Windows.  Visual  Basic  2.0  for 
Windows  is  an  object-oriented  programming  language  that 
incorporates  many  of  the  graphical  user  interface  features 
tested  in  this  experiment.  The  program  was  compiled  into  an 
executable  file  and  run  under  Windows  3.1. 

Data  Collection 

For  the  purpose  of  our  research,  both  quantitative  and 
qualitative  information  were  collected.  The  quantitative 
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information  was  collected  through  the  computer.  A  program 
was  written  by  the  researchers  to  track  total  task 
completion  times  and  total  key  stroke  counts  for  each 
subtask.  Both  measures  began  on  the  first  key  stroke  on  the 
first  screen  for  each  subtask,  and  ended  on  the  last  screen 
( See  Appendices  E  and  F ) . 

Additionally,  we  were  able  to  replay  any  subject's  task 
on  the  screen  because  the  data  being  collected  were  kept  on 
file  in  the  computer.  This  allowed  us  to  replicate  the 
subject's  task  and  more  closely  examine  the  progress  and 
timing  of  each  keystroke. 

The  qualitative  information  was  collected  by  surveying 
the  subjects  after  each  subtask  using  a  questionnaire  (See 
Appendices  C  and  D).  The  questionnaires  combined  close- 
ended  questions,  using  a  5-point  Likert  scale,  and  open- 
ended  questions.  After  each  main  task,  the  subjects  were 
asked  both  closed  and  open-ended  questions  about  all  three 
subtask  features.  Additionally,  the  subjects  were  asked  to 
rank  order  the  access /navigational  features  to  determine 
their  preference. 

Investigative  Questions 

The  data  collected  were  used  to  answer  the  following 
investigative  questions : 

1.  What  are  the  differences  in  total  task  completion 
times  among  the  three  features  in  each  main  task? 
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2.  What  are  the  differences  in  total  key  strokes  among 
the  three  features  in  each  main  task? 

3.  What  is  the  correlation  between  total  task 
completion  times  and  total  key  strokes? 

4.  What  is  the  correlation  between  total  task 
completion  times  and  user  satisfaction? 

5.  What  is  the  correlation  between  total  key  strokes 
and  user  satisfaction? 

6.  Which  access/navigational  feature  provides  the  most 
user  satisfaction? 

7.  What  is  the  statistical  difference  between  non- 
experienced  and  experienced  computer  users  when  rating  the 
features  according  to  user  satisfaction? 

The  answers  to  the  investigative  questions  will 
determine  which  access /navigational  feature  provides  the 
highest  degree  of  user  satisfaction. 

Method  of  Analysis 

Answers  to  the  closed-ended  questions  were  converted  to 
a  number  from  one  to  five  based  on  the  subject's  response  on 
the  5-point  Likert  scale.  The  method  used  to  analyze  the 
data  was  a  one-way  Analysis  of  Variance  (ANOVA) .  We  used 
the  computer  program  STATISTIX*  Version  4.0,  Analytical 
Software,  to  create  a  AMOVA  suxnmary  table  to  determine  the 
variability  attributable  to  the  treatments,  blocks  and 
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errors.  The  ANOVA  summary  table  tested  the  null  hypothesis 
that  there  are  no  differences  among  the  means  for  the  three 
access /navigational  features.  The  Bonferroni  Pairwise- 
Comparison  of  Means  was  used  to  determine  homogeneous  groups 
among  the  feature.  Additionally,  computer  experience  was 
blocked  to  determine  if  computer  experience  affects  the 
means.  An  overall  F-test  with  a  0.05  significance  level  was 
used  to  test  for  statistical  significance. 

Summary 

In  this  chapter,  we  defined  our  methodology.  A 
randomized  block  experimental  design  was  used  to  determine 
the  best  access/navigational  feature.  The  experiment  was 
divided  into  two  main  tasks.  These  tasks  involved  accessing 
information  and  navigating  through  several  computer  screens. 
Personnel,  located  at  Wright-Patterson  AFB,  were  the  test 
subjects  for  this  experiment.  They  were  trained  prior  to 
each  subtask  and  were  asked  questions  relating  to  their 
satisfaction  with  each  access/navigational  feature.  Access 
times  and  total  keystrokes  were  collected  by  the  computer 
program.  This  information,  combined  with  the  questionnaire 
results,  was  used  to  answer  the  investigative  questions.  An 
analysis  of  variances  (ANOVA),  with  a  significance  level  of 
0.05,  was  used  to  analyze  the  data.  The  results  of  the 
experiment  are  presented  in  Chapter  IV,  Findings  and 
Analysis . 
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IV.  Findings  and  Analysis 


Introduction 

As  stated  in  Chapter  3,  this  experiment  used 
quantitative  and  qualitative  measurements.  The  quantitative 
measurements  included  questionnaire  results,  total  task 
completion  times,  and  key  strokes.  The  results  from  the 
questionnaire  were  converted  to  numerical  ratings  of  1-5 
(1  «  strongly  agree,  2  »  agree,  3  »  undecided,  4  »  disagree, 
and  5  =  strongly  disagree)  and  analyzed  using  an  Analysis  of 
Variance  and  Bonferroni  Pairwise  Comparison  of  Means.  These 
data  are  discussed  in  the  Quantitative  Analysis  Section. 

The  mean  values  for  the  total  task  completion  time  and  key 
stroke  measurements  for  each  tested  feature  were  collected 
and  analyzed  using  the  Bonferroni  comparison.  These  data 
are  discussed  in  the  Investigative  Question  Section. 

The  qualitative  measurements  were  collected  using  open- 
ended  questions  on  the  questionnaire.  The  data  collected 
from  these  questions  were  used  to  provide  insight  and 
further  substantiate  the  quantitative  results.  These  data 
are  discussed  in  the  Qualitative  Analysis  Section.  This 
chapter  is  divided  into  three  main  sections.  The  first 
section  discusses  the  quantitative  analysis  for  each  main 
task.  Task  1  and  Task  2.  The  second  section  answers  the 
seven  investigative  questions  posed  in  Chapter  1  and  3.  The 
third  section  addresses  the  qualitative  analysis. 
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summarizing  the  open-ended  questions.  This  chapter 
concludes  with  a  summary  of  our  findings  and  analysis. 


Quantitative  Analysis 

This  section  analyzes  the  quantitative  measurements 
(numerical  responses  to  questionnaire)  collected  in  the 
experiment.  The  analysis  is  presented  in  two  main  sections: 
Task  1  and  Task  2. 

Task  1: _ Screen  Access  and  Navigation.  The  features  in 

this  task  included:  dedicated/hardware  keys,  programmed>le 
soft  keys,  and  push  button  keys.  Each  question  from  the 
questionnaire  is  discussed  separately. 

Question  1.  The  feature  provided  a  high  degree  of 
user  satisfaction  when  moving  from  one  screen  to  the  next, 
Qut  of  28  responses,  the  following  rated  the  feature 
strongly  agree  or  agree:  27  for  the  dedicated/hardware 
keys,  16  for  the  programmable  soft  keys,  and  only  8  for  the 
push  button  keys.  Equally  important  is  the  nximber  of  people 
who  rated  the  feature  strongly  disagree  and  disagree. 
Eighteen  people  rated  the  push  button  keys  this  way.  These 
results  indicate  that  people  prefer  dedicated/hardware  keys 
and  do  not  prefer  push  button  keys  when  moving  from  one 
screen  to  the  next  ( See  Figure  3 ) . 

To  further  analyze  the  results,  we  performed  an 
analysis  of  variances  (ANQVA)  to  determine  if  there  were 


significant  differences  among  the  means  for  the  three 
features.  The  ANOVA  results  are  summarized  in  TeUale  4. 


Figure  3.  Histogram  Depicting  Results 
for  Task  1,  Question  1 


TABLE  4 

RESULTS  OF  THE  ANOVA  FOR  TASK  1,  QUESTION  1 


Source 

n 

Between 

2 

54.0238 

Within 

81 

102.678 

Total 

83 

156.702 

Feature 

Mean 

Ded/Hardware 

1.5714 

Programmable 

2.5714 

Push  Button 

3.5357 

1.2889 _ I _ 

1.3466  I 


To  determine  if  there  is  a  significant  difference  among 
the  means,  we  tested  the  null  hypothesis  that  no  significant 
differences  exist.  If  the  p-value  is  less  than  the  level  of 
significance,  then  the  null  hypothesis  is  rejected  and  the 
alternative  hypothesis,  there  are  significant  differences 
among  the  means,  is  accepted.  Since  the  above  p-value  of 
0.0000  is  less  than  the  level  of  significance  of  0.05,  the 
overall  F-test  shows  that  there  are  significant  differences 
among  the  means.  Table  4  also  summarizes  the  mean  and 
standard  deviation  for  each  feature.  The  overall  F-test 
indicates  that  there  are  “some  contrast  that's  significant, 
but  does  not  guarantee  than  any  pairwise  comparison  is 
particularly  important"  (Statistix  User  Manual,  1992:  203). 

To  determine  if  there  are  any  significant  differences 
among  the  pairwise  comparisons,  we  compared  the  mean  values 
by  using  the  Bonferroni  Pairwise  Comparison  of  Means. 
Bonferroni  results  in  an  experimentwise  error  rate  of  less 
than  a  (0.05).  Bonferroni  is  a  conservative  test  for 
pairwise  comparisons.  Less  conservative  comparison  methods 
may  yield  results  showing  differences,  but  they  result  in 
experimentwise  errors  greater  than  a  (Statistix  User  Manual, 
1992:  203).  This  comparison  shows  homogeneous  groups, 
represented  by  the  I  in  the  same  column.  If  the  groups  are 
not  homogeneous,  the  Is  appear  in  separate  columns. 

According  to  the  results  shown  in  Table  4,  all  three  means 
are  significantly  different  from  one  another — no  two 
features  have  Is  in  the  same  column. 
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since  the  mean  values  are  all  significantly  different 
and  the  mean  value  for  dedicated/hardware  keys  is  1.5714 
(corresponding  to  agree/strongly  agree),  dedicated/hardware 
keys  provided  the  most  user  satisfaction  when  moving  from 
one  screen  to  the  next. 

Question  2 .  The  feature  provided  a  high  degree  of 
user  satisfaction  when  navigating  up  and  down  the  screen. 

Out  of  28  responses,  the  following  rated  the  feature 
strongly  agree  or  agree:  25  for  dedicated/hardware  keys,  15 
for  programmable  soft  keys,  and  6  for  push  button  keys. 
Nineteen  people  rated  push  button  keys  strongly  disagree  or 
disagree.  These  results,  as  shown  in  Figure  4,  indicate 
that  people  prefer  dedicated/hardware  keys  and  do  not  prefer 
push  button  keys  when  navigating  up  and  down  the  screen. 

The  ANOVA  results  are  summarized  in  Table  5. 
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Figure  4.  Histogram  Depicting  Results 


for  Task  1,  Question  2 


TABLE  5 

RESULTS  OF  THE  ANOVA  FOR  TASK  1,  QUESTION  2 


Source 

SS 

MS 

F 

P 

Between 

2 

52.4523 

26.2261 

18.78 

0.0000 

Within 

81 

113.107 

1.39638 

Total 

83 

165.559 

Feature 

Mean 

Std  Dev 

HoaK>ge 

neous  C 

roups 

Oed/Hardware 

1.7142 

0.7629 

I 

Programmable 

2.5357 

1.4005 

I 

Push  Button 

3.428 

1.1816 

I 
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Since  the  above  p-value  of  0.0000  is  less  than  the 
level  of  significance  of  0.05/  the  overall  F-test  shows  that 
there  are  significant  differences  eunong  the  means.  The 
Bonferroni  Comparison  of  Means  shows  that  all  three  means 
are  significantly  different  from  one  another. 

Since  the  mean  values  are  all  significantly  different 
and  the  mean  value  showing  the  highest  degree  of  user 
satisfaction  is  1.7142  for  the  dedicated/hardware  keys,  the 
dedicated/hardware  keys  provided  the  most  user  satisfaction 
when  navigating  up  and  down  the  screen. 

Question  3 .  The  feature  provided  a  high  degree  of 
user  satisfaction  when  moving  from  one  line  to  the  next, 
after  inputting  text.  Out  of  28  responses,  the  following 
rated  the  feature  strongly  agree  or  agree:  27  for 
dedicated/hardware  keys,  15  for  programmable  soft  keys,  and 
9  for  push  button  keys.  Eighteen  people  rated  push  button 
keys  strongly  disagree  or  disagree.  These  results,  as  shown 
in  Figure  5,  indicate  that  people  prefer  dedicated/hardware 
keys  and  do  not  prefer  push  button  keys  when  moving  from  one 
line  to  the  next,  after  inputting  text.  The  ANOVA  results 
are  summarized  in  Table  6. 
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Figure  5.  Histogram  Depicting  Results 


for  Task  1,  Question  3 


TABLE  6 

RESULTS  OF  THE  ANOVA  FOR  TASK  1,  QUESTION  3 


Source 

SS 

NS 

p 

Between 

2 

60.0952 

30.0476 

22.00 

0.0000 

Within 

81 

110.607 

1.36552 

Total 

83 

170.702 

Feature 

Mean 

Std  Dev 

BHiS 

neoua  C 

roups 

Ded/Hardware 

1.5357 

0.5762 

I 

Programmable 

2.5357 

1.4005 

I 

Push  Button 

3.6071 

1.3427 

I 

46 


Since  the  above  p-value  of  0.0000  is  less  than  the 
level  of  significance  of  0.05/  the  overall  F-test  shows  that 
there  are  significant  differences  among  the  means.  The 
Bonferroni  Comparison  of  Means  shows  that  all  three  means 
are  significantly  different  from  one  another. 

Since  the  mean  values  are  all  significantly  different 
and  the  mean  value  showing  the  highest  degree  of  user 
satisfaction  is  1.5357  for  the  dedicated/hardware  keys,  the 
dedicated/hardware  keys  provide  the  most  user  satisfaction 
when  moving  from  one  line  to  the  next/  after  inputting  text. 

Question  4.  Few  accessing  and  selection  errors 
were  made  when  using  this  feature.  Out  of  28  responses/  the 
following  rated  the  feature  strongly  agree  or  agree:  25  for 
dedicated/hardware  keyS/  18  for  programmable  soft  keys,  and 
6  for  push  button  keys.  Eighteen  people  rated  push  button 
keys  strongly  disagree  or  disagree.  These  results,  as  shown 
in  Figure  6,  indicate  that  people  made  the  fewest  access  and 
selection  errors  with  the  dedicated/hardware  keys  and  the 
most  errors  with  the  push  button  keys.  The  ANOVA  results 
are  summarized  in  Table  7 . 
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Figure  6.  Histogram  Depicting  Results 


for  Task  1,  Question  4 


TABLE  7 

RESULTS  OF  THE  ANOVA  FOR  TASK  1,  QUESTION  4 


Source 

n 

SS 

MS 

P 

Between 

2 

45.0238 

22.5119 

17.98 

0.0000 

Within 

81 

101.392 

1.25176 

Total 

83 

146.416 

Feature 

Mean 

Std  Dev 

neous  C 

roups 

Ded/Hardware 

1.8214 

0.9048 

I 

Programmable 

2.3571 

1.2236 

I 

Push  Button 

3.5714 

1.1996 

I 

Since  the  eUsove  p-value  of  0.0000  is  less  than  the 
level  of  significance  of  0.05,  the  overall  F-test  shows  that 
there  are  significant  differences  among  the  means.  The 
Bonferroni  Comparison  of  Means  shows  that  dedicated/hardware 
and  programmable  soft  keys  are  not  significantly  different; 
however,  both  are  significantly  different  from  push  button 
keys . 

Since  dedicated/hardware  and  programmable  soft  keys 
have  the  lowest  mean  value  and  are  significantly  different 
from  push  button  keys,  both  dedicated/hardware  and 
programmable  soft  keys  provided  the  fewest  access  and 
selection  errors. 

Question  5.  Overall,  the  feature  provided  a  high 
degree  of  user  satisfaction  when  accessing  and  navigating 
the  screens.  Out  of  28  responses,  the  following  rated  the 
feature  strongly  agree  or  agree:  25  for  dedicated/hardware 
keys,  16  for  programmable  soft  keys,  and  5  for  push  button 
keys.  Twenty  people  rated  push  button  keys  strongly 
disagree  or  disagree.  These  results,  as  shown  in  Figure  7, 
indicate  that  people  overall  prefer  dedicated/hardware  keys 
and  do  not  prefer  push  button  keys  when  accessing  and 
navigating  the  screen.  The  ANOVA  results  are  summarized  in 
Table  8 . 
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Plgura  7.  Histogram  Depicting  Results 
for  Task  1,  Question  5 


TABLE  8 

RESULTS  OF  THE  ANOVA  FOR  TASK  1,  QUESTION  5 


Source 


Between 


Within 


Total 


Feature 


2 

58.3095 

81 

109.642 

83 

167.952 

Mean 


NS 


29.1547 


1.35361 


21.54  0.0000 


Ded/Hardware 

1.7142 

0.7629 

Programmable 

2.6071 

1.3700 

Push  Button 

3.7500 

1.1634 

Since  the  above  p-value  of  0.0000  is  less  than  the 
level  of  significance  of  0.05,  the  overall  F-test  shows  that 
there  are  significant  differences  aioong  the  means.  The 
Bonferroni  Comparison  of  Means  shows  that  all  three  means 
are  significantly  different  from  one  another. 

Since  the  mean  values  are  all  significantly  different 
and  the  mean  value  showing  the  highest  degree  of  user 
satisfaction  is  1.7142  for  the  dedicated/hardware  keys,  the 
dedicated/hardware  keys  provided  the  most  user  satisfaction 
overall.  The  mean  value  for  push  button  keys  is  3.7500, 
which  corresponds  to  undecided/disagree.  Push  button  keys, 
therefore,  provided  the  least  user  satisfaction  overall. 

Ranked  Questions.  After  each  subject  completed 
Task  1,  using  all  three  features,  they  ranked  the  features 
according  to  user  satisfaction  (1  =  best  and  3  =  worst). 
Each  ranking  question  corresponds  to  the  five  questions 
already  discussed  (R1  =  Question  1).  The  results  of  the 
rankings  are  summarized  in  Figure  8. 
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Screen  Access/Navigation 

Means  of  the  Three  Features 
Ranking  Questions 


0 

r 

R1 

R2 

R3 

_ 

R4 

_ 

R5 

Dedicated  ^ 

1 .3571 

1.2501 

1.2042 

1.2067 

1.2067 

Programmable  S 

1.0671 

1.9205 

1.9206 

1.0671 

1.0920 

Push  button  □ 

2.8571 

2.0928 

29206 

29285 

2.0920 

Figure  8.  Summary  of  the  Mean  Values  for  the 
Rankings  of  the  Three  Features 


Seventy-one  percent  of  the  responses  selected  the 
dedicated/hardware  keys  as  the  best  feature  and  all  but  two 
responses  selected  push  button  keys  as  the  worst  feature. 
The  rankings  support  the  findings  from  Task  1:  The 
dedicated/hardware  keys  provided  the  most  user  satisfaction 
overall  and  the  push  button  keys  provided  the  least  user 
satisfaction  overall.  The  programmable  soft  keys 
consistently  rated  in  the  middle.  Task  2  results  are 
discussed  next. 
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The  features  used 


in  this  task  include:  number  keys,  cursor  control  keys,  and 
programmable  soft  keys.  Each  question  from  the 
questionnaire  is  discussed  separately. 

Question  1.  The  feature  provided  a  high  degree  of 
user  satisfaction  when  moving  from  one  screen  to  the  next. 
Out  of  28  responses,  the  following  rated  strongly  agree  or 
agree:  24  for  the  number  keys,  20  for  the  cursor  control 

keys,  and  19  for  the  programmable  soft  keys.  These  results 
indicate  that  people  were  satisfied  with  all  three  features 
when  moving  from  one  screen  to  the  next  (See  Figure  9). 

To  further  analyze  the  results,  we  performed  an 
analysis  of  variance  (ANOVA)  to  determine  if  there  were 
significant  differences  among  the  means  for  all  three 
features.  The  ANOVA  results  are  summarized  in  Table  9. 
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Figure  9.  Histogram  Depicting  Results  for 
Task  2,  Question  1 


TABLE  9 

RESULTS  OF  THE  ANOVA  FOR  TASK  2,  QUESTION  1 


Source 

n 

SS 

MS 

F 

P  1 

Between 

2 

4.09523 

2.04761 

2.068 

0.1321 

Within 

81 

80.6071 

0.99515 

Total 

83 

84.7023 

Feature 

Mean 

Std  Dev 

HoBoge 

neous  G 

roups 

Nximber  Keys 

1.7500 

0.7993 

I 

Cursor  Keys 

2.1785 

1.0559 

I 

Programmable  Keys 

2.2500 

1.1097 

I 
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To  determine  if  there  is  a  significant  difference  among 
the  means /  we  tested  the  null  hypothesis  that  no  significant 
differences  exist.  If  the  p-value  is  less  than  the  level  of 
significance,  then  the  null  hypothesis  is  rejected  and  the 
alternative  hypothesis,  there  are  significant  differences 
among  the  means,  is  accepted.  Since  the  above  p-value  of 
0.1321  is  greater  than  the  level  of  significance  of  0.05, 
the  overall  F-test  shows  that  there  are  no  significant 
differences  eunong  the  means.  Table  9  also  summarizes  the 
mean  and  standard  deviation  for  each  feature.  We  compared 
the  mean  values  by  using  the  Bonferroni  Pairwise  Comparison 
of  Means.  This  comparison  shows  homogeneous  groups, 
represented  by  the  I  in  the  same  column.  "f  the  groups  are 
not  homogeneous,  the  Is  appear  in  separate  columns. 

According  to  the  results  shown  in  Table  9,  all  three  means 
are  not  significantly  different  from  one  another — all 
features  have  Is  in  the  same  column. 

Since  the  mean  values  correspond  to  agree/strongly 
agree  and  they  are  not  significantly  different,  all  three 
features  provided  a  high  degree  of  user  satisfaction  when 
moving  from  one  main  menu  item  to  the  next. 

Question  2.  The  feature  provided  a  high  degree  of 
user  satisfaction  when  navigating  up  and  down  the  submenu 
items.  Out  of  the  28  responses,  the  following  rated  the 
features  strongly  agree  or  agree:  25  for  number  keys,  19 
for  cursor  control  keys,  and  18  for  programmable  soft  keys. 
These  results  indicate  that  people  were  satisfied  with  all 
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three  features  when  navigating  up  and  down  the  submenu  (See 
Figure  10).  The  ANOVA  results  are  summarized  in  Table  10. 


Figure  10.  Histogram  Depicting  Results  for 
Task  2,  Question  2 


TABLE  10 

RESULTS  OF  THE  ANOVA  FOR  TASK  2,  QUESTION  2 


Source 

SS 

MS 

F 

P 

Between 

2 

5.88095 

2.94047 

3.110 

0.0486 

Within 

81 

76.5357 

0.94488 

Total 

83 

82.4166 

Feature 

Mean 

Std  Dev 

HoMoge 

neous  C 

roup^l 

Number  Keys 

1.7142 

0.7629 

I 

Cursor  Keys 

2.2142 

0.9946 

I 

Programmable  Keys 

2.3214 

1.1239 

I 
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since  the  p-value  of  0.0486  is  less  than  the  level  of 
significance  of  0.05,  the  AMOVA  indicates  that  there  are 
some  differences  among  the  means;  however,  using  Bonferroni 
Pairwise  Comparison  of  Means,  all  three  features  are  not 
significantly  different.  This  inconsistency  exists  because 
the  Bonferroni  comparison  is  the  most  conservative  test  of 
difference  among  means.  We  conclude  that  the  features  are 
not  significantly  different.  The  means  of  all  three 
features  correspond  to  strongly  agree/agree,  therefore,  all 
three  features  provide  a  high  degree  of  user  satisfaction. 

Question  3.  Few  accessing  nd  selection  errors 
were  made  using  this  feature.  Out  of  the  28  responses,  the 
following  rated  the  features  strongly  agree  or  agree:  25 
for  number  keys,  22  for  cursor  control  keys,  and  21  for 
programmable  soft  keys.  These  results  indicate  that  people 
made  few  accessing  and  selection  e;iors  when  using  all  three 
features  (See  Figure  11).  The  ANOVA  results  are  sximmarized 
in  Table  11. 
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Figur9  11.  Histogram  Depicting  Results  for 


Task  2,  Question  3 


TABLE  11 

RESULTS  OF  THE  ANOVA  FOR  TASK  2,  QUESTION  3 


1  Source 

DF 

SS 

MS 

F 

1 

Between 

2 

3.73809 

1.86904 

2.200 

0.1157 

Within 

81 

68.9642 

0.85141 

Total 

83 

72.7023 

1  Feature 

Mean 

Std  Dev 

a 

o 

neous  C 

roups 

Number  Keys 

1.6428 

0.7800 

I 

Cursor  Keys 

2.1071 

0.9560 

I 

Programmable  Keys 

2.0714 

1.0157 

I 
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Since  the  p-value  of  0.1157  is  greater  than  the  level 
of  significance  of  0.05,  the  ANOVA  indicates  that  there  are 
no  differences  among  the  means.  The  Bonferroni  Pairwise 
Comparison  of  Means  shows  that  all  three  features  are  not 
significantly  different.  The  means  of  all  three  features 
correspond  to  strongly  agree/agree,  therefore,  all  three 
features  provide  few  accessing  and  selection  errors. 

Question  4.  Overall,  the  feature  provided  a  high 
degree  of  user  satisfaction  when  accessing  and  navigating 
the  menu.  Out  of  the  28  responses,  the  following  rated  the 
features  strongly  agree  or  agree:  24  for  number  keys,  19 
for  cursor  control  keys,  and  17  for  programmable  soft  keys. 
This  indicates  that  people  experienced  a  high  degree  of  user 
satisfaction  when  using  all  three  features  (See  Figure  12). 
The  ANOVA  results  are  summarized  in  Table  12. 
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Figure  12.  Histogram  Depicting  Results  for 
Task  2,  Question  4 


TABLE  12 

RESULTS  OF  THE  ANOVA  FOR  TASK  2,  QUESTION  4 


Source 


Between 

2 

6.88095 

Within 

81 

87.9285 

Total 

83 

94.8095 

Feature 

Mean 

3.170  0.0461 


3.44047 


1.08553 


Std  Dev  I  HoMogeneous  Groups 


Number  Keys 

1.7142 

0.8099 

I 

Cursor  Keys 

2.3214 

1.1564 

I 

Programmable  Keys 

2.3214 

1.1239 

I 

Since  the  p-value  of  0.0461  is  less  than  the  level  of 
significance  of  0.05,  the  ANOVA  indicates  that  there  are 
some  differences  among  the  means;  however,  using  Bonferroni 
Pairwise  Comparison  of  Means,  all  three  features  are  not 
significantly  different.  This  inconsistency  exists  because 
the  Bonferroni  comparison  is  the  most  conservative  test  of 
difference  among  means.  We  conclude  that  the  features  are 
not  significantly  different.  The  means  of  all  three 
features  correspond  to  strongly  agree/agree,  therefore,  all 
three  features  provide  an  overall  high  degree  of  user 
satisfaction. 

Ranked  Questions.  After  each  subject  completed 
Task  2,  using  all  three  features,  they  ranked  the  features 
according  to  user  satisfaction  (1  =  best  and  3  =  worst). 
Each  ranking  question  corresponded  to  the  four  questions 
already  discussed  (R1  =  Question  1).  The  results  of  the 
rankings  are  summarized  in  Figure  13. 
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r - ^ 

Menu  Access/Navigation 

Means  of  the  Three  Features  - 


R1  R2  R3  R4 


Number  Kays  ^ 

1.3929 

1.4642 

1.5 

1 .3928 

Cursor  Keys  0 

2.3928 

2.2142 

2.3214 

2.3928 

Programmable  Q 

2.1705 

22057 

2.1420 

2.1705 

Figure  13.  Suininary  of  the  Mean  Values  for  the 
Rankings  of  the  Three  Features 


When  forced  to  make  a  distinction  among  the  features, 

61  percent  of  the  subjects  ranked  nximber  keys  the  best. 

There  is  no  clear  distinction  between  cursor  control  keys 
and  programmable  soft  keys.  An  analysis  of  the  variances 
and  the  Bonferroni  Pairwise  Comparison  of  Means  show  a 
significant  difference  between  number  keys  and  the  other  two 
features  (See  Table  13). 
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TABLE  13 


RESULTS  OF  THE  ANOVA  FOR  TASK  2,  RANKINGS 


1  Source 

SS 

MS 

P 

Between 

2 

15.5238 

7.76190 

15.16 

0.0000 

Within 

81 

41.4642 

0.51190 

Total 

83 

56.9880 

Feature 

Mean 

Std  Dev 

Jjtoaoge 

neous  C 

roup^j 

Number  Keys 

1.3928 

0.4973 

I 

Cursor  Keys 

2.3928 

0.7859 

I 

Programmable  Keys 

2.1785 

0.8189 

I 

According  to  the  analysis  in  the  previous  sections,  all 
three  features  provided  a  high  degree  of  user  satisfaction. 
Only  when  the  subjects  compared  the  three  features  did  a 
best  feature,  number  keys,  emerge.  We  conclude  that  number 
keys  provide  the  highest  degree  of  user  satisfaction  when 
accessing  and  navigating  the  menu,  and  there  is  no  clear 
difference  in  user  satisfaction  between  cursor  control  and 
programmable  soft  keys.  The  next  section  addresses  the 
seven  investigative  questions  posed  in  Chapters  1  and  3 . 
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This  section  discusses  and  answers  the  seven 


investigative  questions  posed  in  Chapters  1  and  3. 

Question  1.  What  are  the  differences  in  total  task 
completion  times  among  the  three  features  in  each  main  task. 
Table  14  summarizes  the  ANOVA  results  for  total  task 
completion  times  for  Task  1. 


TABLE  14 

RESULTS  OF  THE  ANOVA  FOR  TOTAL  TASK  COMPLETION  TIMES 

FOR  TASK  1 


Source 

mm 

SS 

MS 

P 

P 

Between 

2 

2.197E05 

1.098E05 

27.16 

0.0000 

Within 

81 

3.276E05 

4044.77 

Total 

83 

5.473E05 

Feature 

Mean 

(Secs) 

Std  Dev 

HoBK>ge 

neous  G 

(roups 

Oed/Hardware 

Keys 

142.75 

51.917 

I 

Programmable 

Keys 

175.02 

47.215 

I 

1  Push  Button  Keys 

263.71 

84.909 

I 

The  ANOVA  and  Bonferroni  comparison  indicate  that  there 
are  significant  differences  among  the  means  for  total  task 
completion  times.  These  results  show  that 
dedicated/hardware  keys  and  programmable  soft  keys  are  the 
same,  but  both  are  different  from  push  button  keys. 
Therefore,  dedicated/hardware  keys  and  programmable  soft 
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keys  provided  the  fastest  total  task  completion  times /  while 
push  button  keys  provided  the  slowest  total  task  completion 
times  for  Task  1. 

Table  15  summarizes  the  ANOVA  results  for  total  task 
completion  times  for  Task  2. 


TABLE  15 

RESULTS  OF  THE  ANOVA  FOR  TOTAL  TASK  COMPLETION  TIMES 

FOR  TASK  2 


1  Source 

E91 

SS 

MS 

P  1 

1  Between 

2 

8415.19 

4207.59 

10.55 

0.0001 

1  Within 

81 

32313.8 

398.936 

f  Total 

83 

40929.0 

Feature 

Mean 

(Secs) 

Std  Dev 

Hoeoge 

neous  G 

rroups 

Number  Keys 

65.402 

17.788 

I 

Cursor  Keys 

89.212 

21.961 

I 

1  Programmable 

Keys 

72.244 

19.951 

I 

The  ANOVA  and  Bonferroni  comparison  indicate  that  there 
are  significant  differences  between  nvunber  and  programmable 
soft  keys  and  cursor  control  keys.  Therefore,  number  and 
programmable  soft  keys  provided  the  fastest  total  task 
completion  times,  while  cursor  control  keys  provided  the 
slowest  total  task  completion  times  for  Task  2. 

To  answer  investigative  question  1,  both  the 
dedicated/hardware  and  programmable  soft  keys  provided  the 
fastest  total  task  completion  times  when  accessing/ 
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navigating  the  screens  and  both  the  number  and  programmable 
soft  keys  provided  the  fastest  total  task  completion  times 
when  accessing/navigating  the  menu. 


Question  2 .  What  are  the  differences  in  total  key 
strokes  among  the  three  features  in  each  main  task? 

Table  16  summarizes  the  ANOVA  results  for  total  key  strokes 
for  Task  1 . 


TABLE  16 

RESULTS  OF  THE  ANOVA  FOR  TOTAL  KEY  STROKES  FOR  TASK  1 


1  Source 

OP 

SS 

MS 

P 

p 

Between 

2 

4738.3 

23769.1 

179.4 

0.0000 

Within 

81 

10734.5 

132.524 

Total 

83 

58272.8 

Feature 

Mean 

Std  Dev 

53BHI^5!85BB 

Ded/Hardware  Keys 

52.392 

8.4210 

I 

Programmable  Keys 

48.285 

4.9729 

I 

Push  Button  Keys 

100.67 

17.376 

I 

The  ANOVA  and  Bonferroni  comparison  indicate  that  there 
are  significant  differences  among  the  means  for  total  key 
strokes.  These  results  show  that  dedicated/hardware  keys 
and  programmable  soft  keys  are  the  same,  but  both  are 
different  from  push  button  keys.  Therefore,  dedicated/ 
hardware  keys  and  programmable  soft  keys  required 
the  least  amount  of  key  strokes,  while  push  button  keys 
required  the  greatest  amount  of  key  strokes  for  Task  1. 
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TeU}le  17  summarizes  the  ANOVA  results  for  total  key 


strokes  for  Task  2. 


TABLE  17 

RESULTS  OF  THE  ANOVA  FOR  TOTAL  KEY  STROKES  FOR  TASK  2 


Source 

n 

SS 

MS 

P 

P 

Between 

2 

13862.9 

6931.47 

722.7 

0.0000 

Within 

81 

776.857 

9.59082 

Total 

83 

14639.8 

Feature 

Mean 

Std  Dev 

HoBK>9e 

neous  Groups 

Number  Keys 

10.714 

1.3569 

I 

Cursor  Keys 

37.785 

5.1377 

I 

Programmable  Keys 

10.357 

0.7310 

I 

The  ANOVA  and  Bonferroni  comparison  indicate  that  there 
are  significant  differences  between  number  and  programmable 
soft  keys  and  cursor  control  keys.  Therefore,  number  and 
progreunmable  soft  keys  required  the  least  amount  of  key 
strokes,  while  cursor  control  keys  required  the  greatest 
amount  of  key  strokes  for  Task  2. 

To  answer  investigative  question  2,  both  the 
dedicated/hardware  and  programmable  soft  keys  required  the 
least  amount  of  total  key  strokes  when  accessing/navigating 
the  screens  and  both  the  number  and  programmable  soft  keys 
the  least  amount  of  total  key  strokes  when  accessing/ 
navigating  the  menu. 
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Question  3.  What  is  the  correlation  between  total  task 
completion  times  and  total  key  strokes?  The  correlation 
between  total  task  completion  times  and  total  key  strokes 
for  Task  1  is  positive,  with  r  *  .7226.  Correlations  range 
from  1  to  -1,  with  zero  indicating  no  correlation.  Since 
.7226  is  close  to  1,  we  conclude  there  is  a  high  positive 
correlation  between  total  task  completion  times  and  total 
key  strokes.  For  Task  2,  the  correlation,  r,  is  0.4951. 

This  correlation  is  not  as  strong  as  Task  1,  but  is  still 
significant.  We  conclude  there  is  a  positive  correlation 
between  total  task  completion  times  and  total  key  strokes 
for  Task  2. 

Question  4.  What  is  the  correlation  between  total  task 
completion  times  and  user  satisfaction?  Using  the  results 
from  the  questionnaire  to  obtain  user  satisfaction  and  using 
the  total  task  completion  times.  Table  18  summarizes  the 
correlations  for  the  ranking  questions. 


TABLE  18 

SUMMARY  QF  CQRRELATIQNS  BETWEEN  TQTAL  TASK  CQMPLETIQN  TIMES 

AND  USER  SATISFACTIQN 


1  TASK  1 

1 

1  Question: 

Q1 

Q2 

Q3 

Q4 

Q5  1 

Correlation 

0.5262 

0.5769 

0.6177 

0.5927 

0.5781  1 

TASK  2 

Question: 

Q1 

Q2 

Q3 

Q4 

Correlation 

0.2133 

0.1797 

0.1497 

0.2133 
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Table  18  indicates  that  for  Task  1,  there  is  a 
correlation  between  total  task  completion  times  and  user 
satisfaction.  We  conclude  that  the  subjects  prefer  features 
with  the  fastest  task  completion  times  when  accessing/ 
navigating  the  screens. 

For  Task  2,  the  table  indicates  that  there  is  a  weak 
correlation  between  total  task  completion  times  and  user 
satisfaction. 

Question  5.  What  is  the  correlation  between  total 
keystrokes  and  user  satisfaction?  Using  the  results  from 
the  questionnaire  to  obtain  user  satisfaction  and  using  the 
total  key  strokes/  the  Table  19  summarizes  the  correlations 
for  the  ranking  questions. 


TABLE  19 

SUMMARY  OF  CORRELATIONS  BETWEEN  TOTAL  KEY  STROKES 
AND  USER  SATISFACTION 


TASK  1 

Question: 

Q1 

Q2 

Q3 

Q4 

Q5 

Correlation 

0.6501 

0.6708 

0.6833 

0.7106 

0.6708 

TASK  2 

Question: 

Q1 

Q2 

Q3 

Q4 

Correlation 

0.3105 

0.1868 

0.2580 

0.3105 

Table  19  indicates  that  for  Task  1,  there  is  a 
correlation  between  total  key  strokes  and  user  satisfaction. 
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We  conclude  that  the  subjects  prefer  features  with  the  least 
amount  of  key  strokes  when  accessing/navigating  the  screens. 

For  Task  2,  Table  19  indicates  that  there  is  a  weak 
correlation  between  total  key  strokes  and  user  satisfaction. 
Therefore,  total  key  strokes  are  not  a  good  indicator  of 
user  satisfaction. 

Question  6.  Which  access/navigation  feature  provides 
the  most  user  satisfaction?  As  discussed  in  the 
Quantitative  Analysis  section,  the  dedicated/hardware  keys 
and  the  number  keys  provided  the  most  user  satisfaction  when 
accessing/navigating  the  screens  and  menus,  respectively. 

Question  7.  What  is  the  statistical  difference  between 
non-experienced  and  experienced  computer  users  when  rating 
the  features  according  to  user  satisfaction?  Table  20 
summarizes  the  ANQVA  results  for  blocking  on  computer 
experience  for  Task  1 . 


TABLE  20 

RESULTS  QF  THE  TASK  1  ANQVA  FQR 
NQN-EXPERIENCED  AND  EXPERIENCED  CQMPUTER  USERS 


Source 

DF 

SS 

MS 

F 

P  1 

Experience 

1 

0.04127 

0.04127 

0.01 

0.92465 

Treatments 

2 

36.1607 

18.0804 

4.204 

0.01798 

Within 

80 

344.047 

4.3006 

Total 

83 

380.2490 
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The  ANOVA  tests  the  null  hypothesis  that  there  are  no 


significant  differences  between  non-experienced  and 
experienced  computer  users  when  rating  the  features  for  user 
satisfaction.  Since  the  p-value,  0.92465,  for  the 
experience  blocking  factor  is  greater  than  the  level  of 
significance,  0.05,  we  fail  to  reject  the  null  hypothesis. 
Therefore,  computer  experience  had  no  effect  on  user 
satisfaction  in  Task  1. 

Table  21  summarizes  the  ANOVA  results  for  blocking 
computer  experience  for  Task  2. 


TABLE  21 

RESULTS  OF  THE  TASK  2  ANOVA  FOR 
NON-EXPERIENCED  AND  EXPERIENCED  COMPUTER  USERS 


1  Source 

SS 

MS 

P 

P 

Experience 

1 

0.11288 

0.11288 

0.027 

0.86988 

Treatments 

2 

8.64285 

4.3214 

1.041 

0.35897 

Within 

80 

332.011 

4.1501 

Total 

83 

340.7667 

Since  the  p-value,  0.86988,  for  the  experience  blocking 
factor  is  greater  than  the  level  of  significance,  0.05,  we 
fail  to  reject  the  null  hypothesis.  Therefore,  computer 
experience  had  no  effect  on  user  satisfaction  in  Task  2. 
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All  subjects  completed  a  questionnaire  evaluating  the 
access/navigational  features  for  each  main  task.  As  stated 
earlier,  the  subjects  rated,  ranked  and  answered  open-ended 
questions  about  each  feature.  This  section  summarizes  the 
open-ended  comments.  Task  1,  screen  access  and  navigation, 
evaluated  dedicated/hardware,  programmable  soft,  and  push 
button  keys. 

Dedicated /Hardware  Keys.  Of  the  29  comments  on 
dedicated/hardware  keys,  26  were  positive  and  only  3  were 
negative.  The  comments  had  three  main  themes:  easy,  fast, 
and  familiar.  Thirteen  subjects  stated  that  the 
dedicated/hardware  keys  were  easy  to  use  and  understand. 
“It's  easy,  straight  forward,  and  less  confusing  than  the 
other  features."  Ten  subjects  commented  that  the  keys  were 
fast  and  familiar.  "Easiest  and  most  familiar  of  the  three 
features . "  The  three  negative  comments  stated  that  the 
dedicated/hardware  keys  were  too  slow  or  required  too  many 
key  strokes. 

Programmable  Soft  K»,ys.  Of  the  32  comments  on 
programmable  soft  keys,  16  were  positive  and  16  were 
negative.  Two  main  positive  themes  surfaced:  easy  and  good 
location.  Nine  subjects  stated  that  the  programmable  soft 
keys  were  easy  to  understand  or  use.  "Easy,  like  using 
function  keys  to  select  a  menu  item."  Five  subjects  liked 
^.ne  location  of  the  keys.  "All  keys  needed  were  in  the  same 
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area,  making  it  easy  to  access."  Three  themes  surfaced  in 
the  negative  comments:  unfamiliar  use  of  keys,  slow,  and 
confusing.  Five  subjects  stated  that  the  keys  were  being 
used  for  functions  that  they  normally  are  not  used  for. 
"Using  a  function  key  (programmable  soft  key)  to  go  up  and 
down  is  strange;  function  keys  should  be  used  for  main  menu 
items."  Five  subjects  stated  that  the  keys  were  slow  and 
four  stated  they  were  confusing.  "You  had  to  stop  and  look 
at  the  screen  to  tell  which  key  would  move  the  cursor  in  the 
direction  you  need.  With  an  arrow  key,  you  know  the 
direction  automatically."  "Selection  was  not  automatic;  you 
had  to  think  about  it." 

Push  Button  Keys.  Of  the  29  comments  about  push  button 
keys,  5  were  positive  and  24  were  negative.  Three  subjects 
stated  they  liked  the  way  push  button  keys  highlighted  the 
selection.  "Liked  being  able  to  watch  the  cursor  highlight 
the  area — easier  to  make  selection  with  less  errors."  Three 
themes  surfaced  in  the  negative  comments;  too  many 
keystrokes,  slow,  confusing.  Nine  subjects  commented  that 
the  push  button  keys  required  too  many  key  strokes  to 
accomplish  a  task.  "Excessive  number  of  movements — worst  of 
all  functions."  "Requires  more  steps  to  get  the  task  done." 
Eight  subjects  stated  that  the  push  buttons  were  slow  and 
seven  stated  that  they  were  confusing.  "Too  slow,  too 
complicated. " 

The  qualitative  analysis  for  Task  1  substantiates  the 
findings  in  the  quantitative  analysis  section.  Dedicated/ 
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hardware  keys  provided  the  highest  and  push  button  keys 
provided  the  lowest  degree  o£  user  satisfaction  when 
accessing  and  xiavigating  the  screens.  Task  2,  menu  access 
and  navigation,  evaluated  the  following  features:  number, 
cursor  control,  and  programmable  soft  keys. 

Number  Keys.  Of  the  34  responses,  28  were  positive  and 
6  were  negative.  Fourteen  subjects  stated  that  the  number 
keys  were  easy  to  use  and  understand.  "Numbers  were  easy  to 
use  because  they  corresponded  to  the  selection  nximber  in  the 
menu."  Others  commented  that  the  number  keys  were  fast  and 
familiar.  The  main  reason  people  did  not  like  the  number 
keys  was  because  they  offered  no  visual  feedback  on  the 
selection  made.  “It  didn't  show  which  one  I  selected." 

Cursor  Control  Keys.  Like  number  keys,  subjects  felt 
the  cursor  control  keys  were  easy.  Of  the  18  positive 
responses,  11  subjects  commented  on  their  ease  of  use.  Four 
subjects  commented  on  the  visual  feedback  of  the  cursor 
control  keys.  "I  could  watch  the  screen  respond  to  each 
cursor  movement."  Of  the  15  negative  responses,  9  subjects 
commented  that  the  cursor  control  keys  were  too  slow  and  5 
commented  that  they  required  too  many  key  strokes.  "Not  as 
fast  as  number  or  programmable  soft  keys."  "Too  slow,  have 
to  move  the  cursor  too  much . " 

Programmable  Soft  Kevs.  Subjects  found  the 
programmable  soft  keys  fast  and  easy  to  use.  Of  the  19 
positive  responses,  11  commented  on  the  ease  of  use,  and  5 
commented  on  the  speed.  "Quick  and  easy  to  understand." 


74 


Of  the  15  negative  responses,  6  conunented  on  the  problem  of 
having  to  look  at  the  menu  at  the  top  of  the  screen,  then 
the  function  options  at  the  bottom  of  the  screen,  and  then 
the  actual  key  itself.  Having  to  line  up  all  three  areas 
and  make  a  selection  was  confusing  and  required 
concentration.  "Did  not  like  having  to  look  at  the  bottom 
of  the  screen  to  pick  function  key  titles."  “Had  to  keep 
changing  concentration  from  top  to  bottom  of  screen  for 
functions . " 

The  qualitative  analysis  for  Task  2  substantiate  the 
quantitative  findings.  Number  keys  provided  the  highest 
degree  of  user  satisfaction  when  accessing  and  navigating 
the  menu. 

Swumary 

Both  the  quantitative  and  qualitative  data  collected 
for  Task  1  indicate  that  dedicated/hardware  keys  clearly 
provide  the  highest  degree  of  user  satisfaction.  In  all 
five  questions,  the  subjects  consistently  rated 
dedicated/hardware  keys  as  strongly  agree  and  agree  for  user 
satisfaction  and  fewest  selection  errors.  In  contrast,  push 
button  keys  were  consistently  rated  as  strongly  disagree  or 
disagree.  The  distinction  among  the  three  features  was  more 
evident  when  the  subjects  were  asked  to  compare  the  three 
features  and  rank  them  according  to  user  satisfaction. 
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Dedicated/hardware  keys  clearly  emerged  as  the  best  feature 
and  push  button  keys  as  the  worst  feature. 

In  the  qualitative  analysis,  many  subjects  stated  that 
they  preferred  the  dedicated/hardware  keys  because  they  were 
fast,  easy,  and  familiar.  This  is  substantiated  by  the  high 
correlation  between  total  task  completion  times  and  user 
satisfaction  and  the  high  correlation  between  total  key 
strokes  and  user  satisfaction.  This  also  explains  why  push 
button  keys  were  not  highly  rated  because  they  had  the 
highest  task  completion  times  and  most  key  strokes. 

In  Task  2,  no  clear  distinction  emerged  among  the  three 
features.  All  three  features  had  mean  values  corresponding 
to  the  strongly  agree  or  agree  response.  This  indicates 
that  all  three  features  provided  a  high  degree  of  user 
satisfaction.  However,  when  asked  to  compare  the  three 
features  against  one  another,  number  keys  emerged  as  the 
best  feature.  We  conclude  that  any  of  the  three  features 
can  provide  a  high  degree  of  user  satisfaction,  but  the 
number  keys  are  the  preferred  feature. 

The  qualitative  analysis  indicates  that  subjects  liked 
the  number  keys  because  they  were  quick.  However,  the 
quantitative  analysis  shows  that  there  is  a  weak  correlation 
between  total  key  strokes/task  completion  times  and  user 
satisfaction.  We  conclude  that  the  number  keys  were  rated 
the  best  feature  because  they  had  few  negative 
characteristics.  Cursor  control  and  programmable  soft  keys 
had  significant  negative  characteristics.  Cursor  control 
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keys  were  too  slow  and  required  too  many  keystrokes,  while 
programmable  soft  keys  required  the  users  to  focus  their 
attention  on  too  many  areas  of  the  screen  and  keyboard. 

There  were  no  significant  differences  between  non- 
experienced  and  experience  computer  users  when  rating  the 
features  for  user  satisfaction.  The  results  of  the 
experiment  clearly  indicate  that  dedicated/hardware  keys  and 
number  keys  provided  the  highest  degree  of  user  satisfaction 
for  their  respective  tasks  for  both  non-experienced  and 
experienced  computer  users. 


As  stated  in  Chapters  1  and  2,  the  Department  of 
Defense  is  incorporating  portable  maintenance  aids  in 
existing  and  emerging  weapon  system  programs.  PMAs  are 
needed  to  access  maintenance  and  technical  information  about 
a  weapon  system.  Armstrong  Leiboratory  (AL)  personnel  design 
maintenance-support  prototypes  for  the  Air  Force,  including 
PMAs.  AL  personnel,  when  designing  the  PMA,  decided  to  use 
a  graphical  user  interface  for  reasons  stated  in  Chapter  2 . 
The  current  PMA  prototype  has  redundant  access  and 
navigational  features.  The  objective  of  this  research  was 
to  evaluate  access/navigational  features  of  a  graphical  user 
interface  installed  on  a  PMA  to  find  the  feature  that 
provided  the  highest  degree  of  user  satisfaction. 

To  test  the  objective  an  experiment  was  conducted.  The 
experiment  asked  subjects  to  complete  two  main  tasks,  screen 
and  menu  access  and  navigation,  and  evaluate  the  features 
according  to  user  satisfaction.  Chapter  4  analyzes  the 
results  of  the  experiment.  This  chapter  is  divided  into 
four  main  sections:  Summary  of  Findings,  Factors  Affecting 
the  Results,  Implications  of  Results,  and  Recommendations 
for  Follow-on  Research.  The  chapter  concludes  with  a  brief 
summary  of  the  thesis  research. 
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Summary  of  Findings 

The  responses  to  the  questionnaire  were  emalyzed  in 
Chapter  4,  Findings  and  Analysis.  Table  22  summarizes  the 
results  of  the  four  questions  relating  to  the  overall  user 
satisfaction  (See  Appendices  C  and  D) . 


TABLE  22 

SUMMARY  OF  THE  RESULTS  OF  EXPERIMENT  FOR  THE 
FOUR  QUESTIONS  RELATING  TO  OVERALL  USER  SATISFACTION 


1  Tas 

k  1:  Screen  Access /Navigation  | 

■ 

p-value 

Test 

Result 

Homogenous  groups 

Best  Feature 

Q5 

0.0000 

Reject 

All  are 

significantly 

different 

Ded/ Hardware 
Keys 

R5 

0.0000 

Reject 

All  are 

significantly 

different 

Ded/Hardware 

Keys 

Tas 

k  2:  Menu  Access /Navigation  | 

■ 

p— value 

Test 

Result 

Homogenous  groups 

Best  Feature 

Q4 

0.0461 

Fail  to 
Reject 

No  significant 
differences  exists 

All  features 
provide  high 
degree  of  user 
satisfaction 

R4 

0.0000 

Reject 

Number  Keys  are 
significantly 
different  from 
cursor  and 
programmable  soft 
keys 

Number  Keys 
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The  results  were  tested  for  significant  differences 
using  an  analysis  of  variance  and  Bonferroni  Pairwise 
Comparison  of  Means.  The  comparison  tests  the  null 
hypothesis  that  no  significant  differences  exists  among  the 
means.  The  p-value  is  the  probability  that  the  null 
hypothesis  is  true. 

As  Table  22  indicates/  the  best  feature  for  screen 
access  and  navigation  is  the  dedicated/hardware  keys.  Both 
question  5  (Q5)  and  its  associated  ranking  question  (RS), 
show  that  all  three  features  are  significantly  different 
with  the  dedicated/hardware  keys  having  the  lowest  mean 
value.  The  mean  value  for  dedicated/hardware  keys  for 
question  5  was  1.7172/  which  corresponds  to  the  strongly 
agree  and  agree  rating  for  high  user  satisfaction. 
Additionally/  the  mean  value  for  the  dedicated/hardware  keys 
for  the  associated  ranking  question  was  1.2857.  This 
indicates  that  a  significant  portion  of  the  subjects  rated 
this  feature  the  best. 

The  results  for  the  menu  access  and  navigation  task  had 
conflicting  results.  When  asked  to  rate  the  features 
strictly  on  user  satisfaction,  there  were  no  significant 
differences,  all  three  features  provided  a  high  degree  of 
user  satisfaction.  All  three  mean  values  were  between  1.7 
and  2.32.  These  values  correspond  to  strongly  agree  and 
agree  ratings  for  high  user  satisfaction.  However,  when 
subjects  ranked  the  three  features,  significant  differences 
emerged.  For  this  ranking  question,  the  mean  value  for 
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number  keys  was  1.3928,  indicating  that  a  significant 
portion  of  the  subjects  rated  this  feature  the  best. 

Factors  Affecting  the  Results 

Several  factors  affected  the  results  of  the  experiment, 
including  the  laptop  computer,  computer  program, 
environment,  learning  curve,  and  screen  information.  We 
used  a  laptop  computer  instead  of  the  prototype  PMA  because 
of  the  modified  computer  screens  needed  for  the  experiment. 
The  laptop  had  several  differences  from  the  PMA. 

Laptop  Computer.  The  laptop  computer  has  a  fully 
functional  keyboard,  whereas  the  PMA  has  a  limited  keyboard 
(See  Figure  14  and  15).  Currently,  the  PMA  does  not  have 
text  keys  for  inputting  letters.  The  PMA  design  forces  the 
technician  to  type  in  text  by  first  accessing  a  screen  with 
letters,  then  using  a  cursor  to  highlight  the  letter  needed. 
The  technician  keeps  highlighting  letters  until  the  word  is 
spelled.  This  design  is  obviously  slower  than  the  laptop 
computer,  which  has  a  full  alpha-numeric  keyboard.  The 
layout  and  size  of  the  keys  are  also  different,  as  seen  in 
Figures  14  and  15.  On  the  PMA,  the  programmable  soft  keys 
are  farther  apart  and  larger  than  the  programmable  soft  keys 
on  the  laptop  computer.  Both  these  factors  could  have 
affected  the  results  of  the  experiment. 
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Figure  14.  Laptop  Keyboard 


Figure  15.  Location  of  Key  Components  on  the  PMA 
(Eagle  Technology,  1991:3) 
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Computer  Program.  The  computer  program  had  some 
limitations  that  could  have  affected  the  results  of  the 
experiment.  The  computer  program  was  written  in  Visual 
Basic  2.0  for  Windows.  The  program  did  not  allow  the  user 
to  move  laterally  on  two  selection  tasks  in  Task  1,  instead 
users  had  to  use  the  up  or  down  key  to  access  the  items . 
Subjects  found  the  lack  of  lateral  movement  an  annoyance, 
but  since  all  features  were  tested  under  the  same  screens, 
it  should  not  have  affected  the  results  adversely.  Another 
factor  that  could  have  affected  the  results  was  the  speed  of 
the  program  itself.  Subjects  complained  that  the  screen 
speed  was  slow  when  using  cursor  keys.  The  screen  redraw 
rates  were  slow  because  of  software  and  hardware 
limitations.  Also  there  were  some  complaints  that  the 
cursor  keys  were  slow  to  respond  after  some  key  inputs. 

Environment .  The  PMAs  are  slated  to  be  used  on  the 
flightline  for  technicians  to  use  to  access  technical 
information.  The  experiment  took  place  in  a  maintenance 
back  shop,  the  avionics  shop.  This  environmental 
artificiality  may  have  affected  the  experiment  because  we 
did  nut  have  to  deal  with  the  outside  elements. 

Learning  Curve.  The  learning  curve  during  the 
experiment  was  obvious.  After  running  through  one  or  two 
subtasks,  subjects  became  familiar  with  the  computer,  the 
screens,  and  the  experiment.  This  is  known  as  the  recency 
effect.  The  familiarity  gave  them  confidence  and  their 
speed  usually  increased.  We  tried  to  counteract  the  recency 
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effect  by  randomizing  the  task  and  subtask  order,  and 
changing  the  screen  selections  to  minimize  the  risk  of 
subjects  not  reading  the  screen  directions  because  they 
remembered  the  directions  from  the  last  subtask.  However, 
we  do  not  feel  their  opinion  of  the  features  were  affected 
by  the  recency  effect.  On  the  contrary,  subjects  had 
definite  opinions  on  what  features  they  liked,  especially  in 
Task  1.  The  correlation  between  task  order  and  responses 
was  weak  in  both  tasks. 

Screen  Information.  The  researchers  used  actual 
screens  from  the  F/A-18  technical  information  to  develop  the 
screens  for  the  experiment.  AL  uses  F/A-18  technical 
information  on  the  prototype  PMA.  The  screens  were  modified 
for  the  experiment.  In  doing  so,  much  of  the  complex 
information  was  replaced  with  more  simplistic  information. 
The  tasks  were  slightly  modified  to  delete  unnecessary 
screens.  Since  the  subjects  were  evaluating  access/ 
navigational  features  and  not  maintenance  technical  data, 
the  modified  F/A-18  screens  should  have  affected  the 
experiment  only  slightly. 

Despite  the  above  factors,  the  experiment  provides 
information  about  access/navigational  features  that  can  be 
used  in  the  field. 
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Implications  of  Results 


Results  from  the  e3q>eriment  show  that  dedicated/ 
hardware  keys  provide  the  highest  degree  of  user 
satisfaction  when  accessing  and  navigating  screens  and 
number  keys  provide  the  highest  degree  of  user  satisfaction 
when  accessing  and  navigating  the  menu.  Push  button  keys 
and  programmable  soft  keys  provided  the  lowest  degree  of 
user  satisfactions  when  accessing  and  navigating  the  screens 
and  menu,  respectively.  These  results  have  important 
implications  for  future  and  emerging  weapon  systems. 

According  to  AL  personnel,  the  F-22  program  is 
considering  using  a  PMA  that  relies  heavily  on  programmable 
soft  keys  (Masquelier,  1993).  Our  research  does  not  support 
this  decision.  In  Task  1,  screen  access  and  navigation,  the 
programmable  soft  keys  ranked  second  behind  the 
dedicated/hardware  keys.  Subjects  did  not  like  using  the 
progrcunmable  soft  keys  because  their  function  was  not 
consistent  with  what  they  were  already  familiar  with.  In 
Task  2,  menu  access  and  navigation,  the  programmable  soft 
keys  were  ranked  behind  dedicated/hardware  keys.  Subjects 
did  not  like  using  the  programmable  soft  keys  when  accessing 
and  navigating  the  menu  because  they  had  to  focus  their 
attention  on  too  many  areas:  the  menu  itself,  the  function 
option,  and  the  actual  key.  The  research  suggests  that 
dedicated/hardware  and  niunber  keys  should  be  incorporated  on 
future  PMAs,  given  the  options  we  tested. 
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Recommendations  for  Follow-on  Research 


This  experiment  demonstrated  that  dedicated/hardware 
keys  and  number  keys  provide  a  high  degree  of  user 
satisfaction  when  accessing  and  navigating  screens  and 
menus.  However,  there  are  still  many  areas  that  should  be 
tested  before  an  operational  PMA  is  fielded.  Possible 
research  areas  include  the  use  of  different  input  devices  on 
the  PMA.  Possible  input  devices  include  the  mouse,  light 
pen,  trackball,  and  touchscreen.  According  to  AL  personnel, 
the  B-2  program  personnel  are  investigating  the  use  of 
touchscreens  for  their  PMA  (Masquelier,  1993).  The 
different  input  devices  may  affect  how  users  feel  about  the 
six  access/navigational  features  tested  in  this  experiment. 
For  example,  push  buttons  were  rated  the  worst  feature,  but 
might  be  rated  higher  if  the  user  uses  a  touchscreen  or 
mouse.  Additionally,  there  are  other  access/navigational 
features  that  could  be  tested  in  addition  to  the  six  tested 
in  this  experiment. 

Another  possible  area  of  research  is  to  compare  the 
performances  of  maintenance  technicians  when  using  a  PMA 
versus  an  in-shop  workstation,  incorporating  the  same 
interface.  Under  the  IMIS  concept,  the  same  interface  is 
used  on  the  PMA  and  workstation  so  the  user  has  to  learn 
only  one  computer  system.  A  test  to  see  if  performance 
differences  exist  might  result  in  slightly  different 
interfaces  for  the  two  computer  systems. 
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As  Stated  earlier,  there  are  many  factors  that  may  have 
affected  the  results.  Possible  areas  of  future  research 
would  be  to  vary  these  factors  to  test  for  significant 
differences.  One  factor  that  could  be  tested  is  the 
environment.  This  experiment  used  a  controlled  environment. 
However,  the  PMA  is  to  be  used  on  the  flightline,  where  many 
adverse  conditions  exists.  Using  the  same  PMA,  a  comparison 
of  performances  could  be  made  by  testing  the  PMA  features  in 
both  environments — in-shop  and  flightline. 

During  the  experiment,  many  subjects  commented  on  using 
laptops  on  the  flightline.  Many  favored  the  use  of  laptops 
versus  a  PMA  produced  by  the  DoD.  Since  a  prototype  PMA 
exists,  a  comparison  of  technical  information  on  a  laptop 
computer  versus  a  prototype  PMA  could  be  conducted.  The 
implications  from  this  research  might  lead  the  DoD  to 
procure  commercially-available  hardware,  rather  than 
producing  their  own. 

Summary 

In  line  with  the  CALS  initiative,  portable  maintenance 
aids  (PMAs)  are  being  developed  to  store  and  access 
technical  information  on  the  flightline.  The  Armstrong 
Laboratory  located  at  Wright  Patterson  AFB,  OH,  mission  is 
to  design  maintenance  support  prototypes  for  the  Air  Force. 
One  of  the  laboratory's  projects  is  to  develop  a  prototype 
PMA.  The  PMA  incorporates  a  graphical  user  interface  (GUI). 
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The  GUI  was  developed  in  accordance  with  military 
specifications,  the  lETM-GCSFUI.  There  are  access/ 
navigational  features  used  with  the  PMA  that  have  never  been 
tested  for  user  satisfaction. 

Armstrong  Laboratory  personnel  requested  research  be 
performed  to  determine  the  best  access /navigational  feature 
— the  feature  that  provides  the  highest  degree  of  user 
satisfaction.  The  experiment  was  divided  into  two  main 
tasks:  screen  access  and  navigation  evaluating 
dedicated/hardware  keys,  programmable  soft  keys,  and  push 
button  keys;  and  menu  access  and  navigation  evaluating 
number  keys,  cursor  control  keys,  and  programmable  soft 
keys . 

Screens  from  the  prototype  PMA  were  modified  to  test 
the  access/navigational  features  one-at-a-time  using  a 
laptop  personal  computer.  Twenty-eight  maintenance 
technicians  performed  the  experiment.  They  were  asked  to 
rate  and  rank  each  feature  according  to  user  satisfaction. 
The  best  feature  for  screen  access  and  navigation  was  the 
dedicated/hardware  keys  and  the  best  feature  for  menu  access 
and  navigation  was  the  number  keys.  These  results  have 
important  implications  because  the  F-22  program  is  relying 
heavily  on  programmable  soft  keys  on  their  PMAs.  This 
experiment  tested  six  access/navigational  features  installed 
on  the  PMA;  however,  more  research  should  be  conducted 
before  the  DoD/Air  Force  decides  on  a  standard  interface 
design  for  the  PMA. 
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Appendix  A; _ Agr9nYm§ 


ACALS  - 
AFHRL  - 
AIMS  - 

AIP  - 
AL  - 
ANOVA  - 
ATDPS  - 
ATOS  - 
CALS  - 

CAMS  - 
CEMS  - 
CMAS  - 
DOD  - 
EDMICS  - 

GCSFUI  - 

GUI  - 
HMDD- 
lETM  - 
IMIS  - 
J-CALS 
JUSTIS 

MIW  - 


Army  CALS 

Air  Force  Human  Resource  Laboratory 

Advanced  Tactical  Fighter  Integrated  Maintenance 
Information  System 

Aircraft  Interface  Panel 

Armstrong  Laboratory 

Analysis  of  Variances 

Automated  Technical  Data  Presentation  System 

Automated  Technical  Order  System 

Computer-aided  Acquisition  and  Logistic  Support 
System 

Core  Automated  Maintenance  System 
Comprehensive  Engine  Management  System 
Computer-based  Maintenance  Aids  System 
Department  of  Defense 

Engineering  Data  Management  Information  and 
Control  System 

General  Content,  Style,  Format,  and  User- 
Interactions  Requirements 

Graphical  User  Interface 

Head  Mounted  Display  Device 

Interactive  Electronic  Technical  Manuals 

Integrated  Maintenance  Information  System 

Joint-CALS 

Joint  Uniformed  Service  and  Technical  Information 
System 

Maintenance  Information  Workstation 
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PMA  - 
TO  - 


Portable  Maintenance  Aid 
Technical  Orders 
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Appendix  B: _ Sample  Background  Survey 


NAME; 

1.  Check  one;  _  Military  _  Civilian 

_  Male  _  Female 

2 .  Age  _ 

3.  Total  active  military  service  _  yrs  _ _  months 

4.  Current  pay  grade /rank  _ 

5.  Current  job  specialty  or  rating 

6.  Prior  work  experience,  last  two  jobs; 

a)  .  Career /Occupation  specialty _ 

Location  _ 

Inclusive  dates  (m/y)  _ / _  to  _ / _ 

b)  .  Career /Occupation  specialty _ 

Location  _ 

Inclusive  dates  (ra/y)  _ / _  to  _ / _ 

7.  Computer /Electronic  experience; 

a)  .  How  many  years /months  computer  experience  do  you 

have? 

_  yrs  _  months 

b)  .  Have  you  ever  received  formalized  computer 

training?  _ Yes  No 

If  yes,  what  training  and  for  how  long?  _ 


c).  If  you  never  received  formalized  training,  would 
you  consider  yourself  self-taught?  _ Yes  _ ^No 
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If  yes,  what  system  did  you  learn  on?  _ 

d)  .  Do  you  own  your  o%m  conqputer?  _ Yes  _ ^No 

If  yes,  what  model/type?  _ 

e) .  Do  you  use  a  computer  at  work?  _ Yes  _ No 

If  yes,  what  model/type?  _ 

f) .  How  many  hours  a  week  do  you  use  a  computer? 

_  hours 

g)  .  What  do  you  use  the  computer  for  the  majority  of 

the  time?  (ie:  word  processing,  computer  games, 
finances,  spread  sheets,  e-mail  etc.) 


h).  Please  list  the  top  three  programs  you  use  (either 
at  work  or  home)  and  specify  the  operating  systems 
(DOS,  Windows,  MAC,  Amiga,  UNIX  etc.)  Example: 
WordPerfect  for  Windows  or  Word  for  MAC. 

1). _ 

2). _ 

3). _ 


i)- 


Of  the  following  devices 
you  have  personally  used 

keyboard  _ 

numeric  key  pad  _ 

mouse  _ 

light  pen  _ 

touch  screen  _ 

track  ball  _ 

joy  stick  _ 

other,  please  specify 


and  software,  check  those 
and  are  familiar  with: 
text  editor 
word  processor 
file  manager 
electronic  spreadsheet 
electronic  mail 
computer  games 
video  games 


j).  Please  write  any  other  comments  below: 
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Appendix  C:  Task  1  Sample  Questionnaire 


ID  # 


User  Evaluation  Questionnaire 


I .  Questions  fox  Task  1 :  Screen  Access  and  Navigation 


Please  answer  the  following  questions  based  on  your 
participation  in  TASK  1.  After  reading  the  question  circle 
the  number  corresponding  to  your  choice. 
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OlOICATIO/BJUffillMa  KXXS: 


ID  # 


1.  The  dedicated  keys  provided  a  high  degree  of  user  satisfaction  when 
moving  from  one  screen  to  the  next. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


2.  The  dedicated  keys  provided  a  high  degree  of  user  satisfaction  when 
navigating  up  and  down  the  screen. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4 .  Disagree 

5.  Strongly  Disagree 


3.  The  dedicated  keys  provided  a  high  degree  of  user  satisfaction  when 
moving  from  one  line  to  the  next,  after  inputting  text. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


4.  Few  accessing  and  selection  errors  were  made  using  the  dedicated 
keys. 

1.  Strongly  Agree 

2.  Agree 

3.  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


5.  Overall,  the  dedicated  keys  provided  a  high  degree  of  user 
satisfaction  when  accessing  and  navigating  the  screens. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


6.  What  did  you  like  about  this  access  feature? 


7.  What  did  you  not  like  about  this  access  feature? 
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sorr  Kxxs: 


ID  # 


1.  The  programmable  soft  keys  provided  a  high  degree  of  user 
satisfaction  when  moving  from  one  screen  to  the  next. 

1.  Strongly  Agree 

2 .  Agree 

3.  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


2.  The  programmable  soft  keys  provided  a  high  degree  of  user 
satisfaction  when  navigating  up  and  down  the  screen. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4 .  Disagree 

5.  Strongly  Disagree 


3.  The  programmable  soft  keys  provided  a  high  degree  of  user 
satisfaction  when  moving  from  one  line  to  the  next,  after  inputting 
text. 

1.  Strongly  Agree 

2.  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


4.  Few  accessing  and  selection  errors  were  made  using  the  programmable 
soft  keys. 


1.  Strongly  Agree 

2 .  Agree 

3.  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


5.  Overall,  the  programmable  soft  keys  provided  a  high  degree  of  user 
satisfaction  when  accessing  and  navigating  the  screens. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Hndecided 

4.  Disagree 

5.  Strongly  Disagree 


6.  What  did  you  like  about  this  access  feature? 


7.  What  did  you  not  like  about  this  access  feature? 
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ID  # 


F08B  SUTTON  RBXS: 

1.  The  puah  button  keys  provided  a  high  degree  of  user  satisfaction 
when  moving  from  one  screen  to  the  next. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


2.  The  push  button  keys  provided  a  high  degree  of  user  satisfaction 
when  navigating  up  and  down  the  screen. 

1.  Strongly  Agree 

2.  Agree 

3 .  Undecided 

4 .  Disagree 

5.  Strongly  Disagree 


3.  The  push  button  keys  provided  a  high  degree  of  user  satisfaction 
when  moving  from  one  line  to  the  next,  after  inputting  text. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


4.  Few  accessing  and  selection  errors  were  made  using  the  push  button 
keys . 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


5.  Overall,  the  push  button  keys  provided  a  high  degree  of  user 
satisfaction  when  accessing  and  navigating  the  screens. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


6.  What  did  you  like  about  this  access  feature? 


7.  What  did  you  not  like  about  this  access  feature? 
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ID  # 


ZI.  Qiwstions  Bvaliuitlng  All  ShrM  Vttaturas 

Please  answer  the  following  questions  based  on  your  completion  of 
TASK  1.  After  reading  the  question,  rank  the  three  access  features  from 
best  (1)  to  worst  (3) . 


1.  Which  feature  provided  the  highest  degree  of  user  satisfaction  when 
moving  from  one  screen  to  the  next? 

_ Dedicated/Hardware  Keys 

_ Programmable  Soft  Keys 

_ Push  Button  Keys 

2.  Which  feature  provided  the  highest  degree  of  user  satisfaction  when 
navigating  up  and  down  the  screen? 

_ Dedicated/Hardware  Keys 

_ Programmable  Soft  Keys 

_ Push  Button  Keys 

3.  Which  feature  provided  the  highest  degree  of  user  satisfaction  whei 
moving  from  one  line  to  the  next,  after  inputting  text? 

_ Dedicated/Hardware  Keys 

_ Programmable  Soft  Keys 

_ Push  Button  Keys 

4.  Which  feature  caused  the  fewest  access  and  selection  errors? 

_ Dedicated/Hardware  Keys 

_ Programmable  Soft  Keys 

_ Push  Button  Keys 

5.  Overall,  which  feature  provided  the  highest  degree  of  user 
satisfaction  when  accessing  and  navigating  the  screens. 

_ Dedicated/Hardware  Keys 

_ Programmable  Soft  Keys 

_ Push  Button  Keys 
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ID  # 


6.  If  you  have  a  preference,  which  computer  access  feature  did  you 
prefer  and  why? 


7.  Did  your  access  preference  work  best  in  all  task  steps? 
If  no,  which  steps  did  it  not  work  best  in? 


8.  Would  another  computer  access  feature  be  more  practical? 
If  yes,  please  name  the  access  feature. 


9.  What  advantages  would  your  suggested  con^uter  access  feature  (from 
question  #8)  have  over  the  tested  access  features? 


10.  What  recommendations  do  you  have  to  in^rove  future  experiments? 


11.  Do  you  have  any  other  comments  about  this  experiment? 
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Appendix  D;  Task  2  Sample  Questionnaire 


ID  # 


User  Evaluation  Questionnaire 


I .  Questions  for  Task  2 :  Monu  Accoss  and  Navigation 

Please  answer  the  following  questions  based  on  your 
participation  in  TASK  2.  After  reading  the  question  circle 
the  number  corresponding  to  your  choice. 
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ID  # 


MOMBBR  KBXS: 

1.  The  number  beys  provided  a  high  degree  of  user  satisfaction  tdien 
moving  from  one  main  ntenu  item  to  the  next. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


2.  The  number  keys  provided  a  high  degree  of  user  satisfaction  when 
navigating  up  and  down  the  sub-menu  items. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4 .  Disagree 

5.  Strongly  Disagree 


3.  Few  accessing  and  selection  errors  were  made  using  the  number  keys. 

1.  Strongly  Agree 

2.  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


4.  Overall/  the  number  keys  provided  a  high  degree  of  user  satisfaction 
when  accessing  and  navigating  the  menu. 

1.  Strongly  Agree 

2 .  Agree 

3.  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


5.  What  did  you  like  about  the  access  feature? 


6.  What  did  you  not  like  about  the  access  feature? 
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CURSOR  COHTROL  XIXS: 

1.  The  cursor  control  keys  provided  a  high  degree  of  user  satisfaction 
when  moving  from  one  main  menu  item  to  the  next. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4 .  Disagree 

5.  Strongly  Disagree 


2.  The  cursor  control  keys  provided  a  high  degree  of  user  satisfaction 
when  navigating  up  and  down  the  sub-menu  items. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4 .  Disagree 

5 .  Strongly  Disagree 


3,  Few  accessing  and  selection  errors  were  made  using  the  cursor 
control  keys. 

1.  Strongly  Agree 

2 .  Agree 

3.  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


4.  Overall,  the  cursor  control  keys  provided  a  high  degree  of  user 
satisfaction  when  accessing  and  navigating  the  menu. 

1.  Strongly  Agree 

2 .  Agree 

3.  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


5.  What  did  you  like  about  the  access  feature? 


6.  What  did  you  not  like  about  the  access  feature? 
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1.  The  programnable  soft  keys  provided  a  hi^h  degree  of  user 
satisfaction  when  moving  from  one  main  menu  item  to  the  next. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4 .  Disagree 

5.  Strongly  Disagree 


2.  The  programmable  soft  keys  provided  a  high  degree  of  user 
satisfaction  when  navigating  up  and  dom  the  sub-menu  items. 

1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


3.  Few  accessing  and  selection  errors  were  made  using  the  programmable 
soft  keys. 


1.  Strongly  Agree 

2 .  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


4.  Overall,  the  programmable  soft  keys  provided  a  high  degree  of  user 
satisfaction  when  accessing  and  navigating  the  menu. 

1.  Strongly  Agree 

2.  Agree 

3 .  Undecided 

4.  Disagree 

5.  Strongly  Disagree 


5.  What  did  you  like  about  the  access  feature? 


6.  What  did  you  not  like  about  the  access  feature? 
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ZZ.  QiMstlons  Kvaluatlng  All  Thr>»  raAtuzws 

Please  answer  the  following  questions  based  on  your  coiqpletion  of 
tAA  2.  After  reading  the  question,  rank  the  three  access  features  from 
best  (1)  to  worst  (3) . 


t  1.  Which  feature  provided  the  highest  degree  of  user  satisfaction  when 

nusving  from  one  main  menu  item  to  the  next? 

_ ^Number  Keys 

_ Cursor  Keys 

_ Programmable  Soft  Keys 

2.  Which  feature  provided  the  highest  degree  of  user  satisfaction  when 
navigating  up  and  dovm  the  sub-menu  items? 

_ Number  Keys 

_ Cursor  Keys 

_ Programmable  Soft  Keys 

3.  Which  feature  caused  the  fewest  access  and  selection  errors? 

_ ^Number  Keys 

_ Cursor  Keys 

_ Programmable  Soft  Keys 

4.  Overall,  which  feature  provided  the  highest  degree  of  user 
satisfaction  when  accessing  and  navigating  the  menu. 

_ ^Number  Keys 

_ Cursor  Keys 

_ Programmable  Soft  Keys 
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5.  If  you  have  a  preference,  which  coBqputer  access  feature  did  you 
prefer  and  why? 


6.  Did  your  access  preference  work  best  in  all  task  steps? 
If  no,  which  steps  did  it  not  work  best  in? 


7.  Would  another  computer  access  feature  be  more  practical? 
If  yes,  please  name  the  access  feature. 


8.  What  advantages  would  your  suggested  computer  access  feature  (from 
question  #7)  have  over  the  tested  access  features? 


9.  What  recommendations  do  you  have  to  improve  future  experiments? 


10.  Do  you  have  any  other  comments  about  this  experiment? 
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